{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LighTag data post-processing\n",
    "Let's download the dataset from LightTag using their API and postprocess the data + push it to the hub (downloading from the LightTag's UI doesn't work properly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "LIGHTTAG_DOMAIN = 'demo'  #should be your lighttag domain\n",
    "domain = \"bigcodepii\"\n",
    "SERVER = f'https://{domain}.lighttag.io/api/'\n",
    "API_BASE = SERVER +'v1/'\n",
    "MY_USER=\"XX\"\n",
    "MY_PWD=\"XX\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(SERVER+'auth/token/login/',\n",
    "              json={\"username\":MY_USER,\"password\":MY_PWD})\n",
    "assert response.status_code ==200, \"Couldn't authenticate\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "auth_details = response.json()\n",
    "token = auth_details['key']\n",
    "assert auth_details['is_manager'] ==1, \"not a manager\" # Check you are a manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': '87c47b2c-d503-4967-b314-c04d1e7f8be7',\n",
       "  'slug': 'default',\n",
       "  'url': 'http://bigcodepii.lighttag.io/api/v1/projects/default/',\n",
       "  'name': 'default'}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session = requests.session()\n",
    "session.headers.update({\"Authorization\":\"Token {token}\".format(token=token)})\n",
    "#Try it out\n",
    "session.get(API_BASE+'projects/').json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399',\n",
       " 'name': 'PII labeling pre-filtered data',\n",
       " 'slug': 'pii-labeling-pre-filtered-data',\n",
       " 'url': 'http://bigcodepii.lighttag.io/api/v1/projects/default/task_definitions/pii-labeling-pre-filtered-data/',\n",
       " 'allow_suggestions': True,\n",
       " 'annotators_per_example': 1,\n",
       " 'async_status': 'done',\n",
       " 'archived': False,\n",
       " 'priority': 1,\n",
       " 'active': True,\n",
       " 'guidelines': '## Task Overview\\n\\nWelcome to our annotation task. In this task we\\'ll present you with one code file at a time and ask you to tag specific entities. We\\'ll be using this data to evaluate PII detection tools on source code from different programming languages.   \\n\\n1. Please highlight the entire span for each tags where applicable. For example: For tag `NAME`, if the text presented has John Doe, please highlight John Doe as one span, instead of highlighting John and Doe separately.\\n2. If you think a word that should be highlighted, but unsure about which tag to go to, use `AMBIGUOUS` instead.\\n3. **Do not overlap** tags. Tag the one most applicable to the entire span. For example, if a person\\'s name is part of `EMAIL`, do not tag `NAME`.\\n\\n## Tags Guidelines\\n\\nFor each file, please highlight you find any corresponding to the following tags: \\n\\n### API_KEY\\n\\nIncluding API Keys or API Tokens, Bearer Tokens, OAuth Tokens, see [here](https://www.freecodecamp.org/news/best-practices-for-building-api-keys-97c26eabfea9/#:~:text=the%20right%20way.-,API%20Key%20Generation,-Since%20the%20API) for example. Please highlight only the actual key.\\n\\n### AMBIGUOUS\\n\\nIf unsure whether to highlight a text, or which tag to highlight a text for, tag as `AMBIGUOUS` to allow reviewers to take a look later.\\n\\n### EMAIL\\n\\nEmail address, including generic emails such as support@organization.com.\\n\\n### IP_ADDRESS\\n\\nIP addresses could come in two different formats, see [IPv4 and IPv6 address formats](https://www.ibm.com/docs/en/ts3500-tape-library?topic=functionality-ipv4-ipv6-address-formats). Please tag both formats. If unsure a span is an IP address or not, tag `AMBIGUOUS` instead.\\n\\n### NAME\\n\\nA person\\'s name, including just first name or last name only. Do not include title used. For example, if \"Ms. Doe\" is used, please highlight \"Doe\" only.\\nFor now we are not tagging usernames.\\n\\n### PASSWORD\\n\\nAny authentication credentials not applicable to SSH keys or API keys. If unsure, tag `AMBIGUOUS`.\\n\\n### SSH_KEY\\nSecure Shell Key, the output usually looks like `ssh-rsa public_key account`, see example [here](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key). Please highlight the entire public key span. \\n\\n### USERNAME\\n\\nAny username that is used, including for credentials or handles (such as GitHub handles). If username is an email, tag `EMAIL` instead.\\n\\n## Additional note: Class CONTAINS_NON_ENGLISH\\n\\nIf you find a file with comments in a natural language you don\\'t undertsand and can\\'t decide if it includes PII, label it with the class `CONTAINS_NON_ENGLISH`.',\n",
       " 'schema_id': '33a5ef29-c22f-4e64-8fee-6019dde1c76d',\n",
       " 'dataset_id': '5ed27353-49a7-42e9-bc84-81e15f3f4162',\n",
       " 'project_id': '87c47b2c-d503-4967-b314-c04d1e7f8be7',\n",
       " 'created_at': '2022-11-03T17:18:13.282687Z',\n",
       " 'relationSchema_id': None,\n",
       " 'suggestion_models': ['7490d065-5329-4561-8cdb-71f3416033e2'],\n",
       " 'teams': ['9b13d981-9ade-4dd5-8bf2-886c9f11945d'],\n",
       " 'labels': [],\n",
       " 'hide_example_metadata': False}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_definitions = (session.get(API_BASE+'projects/default/task_definitions/').json())\n",
    "task_definitions[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://bigcodepii.lighttag.io/api/v1/projects/default/task_definitions/pii-labeling-pre-filtered-data/'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_td = task_definitions[1]['url'] # Get the url for the test set task definition\n",
    "test_td"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = session.get(test_td+'download/').json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['id', 'examples', 'schema', 'dataset', 'relations', 'name', 'annotators_per_example'])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = data[\"examples\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['content', 'seen_by', 'comments', 'metadata', 'example_id', 'annotations', 'classifications'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total # reviewed annotations: 1318\n",
      "total # annotations: 1318\n"
     ]
    }
   ],
   "source": [
    "# verify that all annotations are reviewed\n",
    "count = 0\n",
    "for example in dataset:\n",
    "    for annotation in example[\"annotations\"]:\n",
    "        assert annotation[\"reviewed\"] == True\n",
    "        count += 1\n",
    "print(f\"total # reviewed annotations: {count}\")\n",
    "print(f\"total # annotations: {sum([len(ex['annotations']) for ex in dataset])}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n"
     ]
    }
   ],
   "source": [
    "for i, e in enumerate(dataset):\n",
    "    if e[\"example_id\"] == \"251926c8-f8cb-443f-99e0-77254d63430d\":\n",
    "        print(i)\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'end': 33732,\n",
       "  'tag': 'PASSWORD',\n",
       "  'start': 33724,\n",
       "  'value': 'password',\n",
       "  'tag_id': 'd7945153-264b-42d8-8db8-e98fa49397a7',\n",
       "  'correct': True,\n",
       "  'reviewed': True,\n",
       "  'example_id': '251926c8-f8cb-443f-99e0-77254d63430d',\n",
       "  'annotated_by': [{'annotator': None,\n",
       "    'timestamp': None,\n",
       "    'annotator_id': None}],\n",
       "  'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399',\n",
       "  'tagged_token_id': '38e747c4-76f3-468e-a660-33c98f79c24a'},\n",
       " {'end': 33372,\n",
       "  'tag': 'PASSWORD',\n",
       "  'start': 33364,\n",
       "  'value': 'wrongpwd',\n",
       "  'tag_id': 'd7945153-264b-42d8-8db8-e98fa49397a7',\n",
       "  'correct': True,\n",
       "  'reviewed': True,\n",
       "  'example_id': '251926c8-f8cb-443f-99e0-77254d63430d',\n",
       "  'annotated_by': [{'annotator': 'shamik.bose89@gmail.com',\n",
       "    'timestamp': '2022-11-05T16:51:02.317+00:00',\n",
       "    'annotator_id': 10}],\n",
       "  'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399',\n",
       "  'tagged_token_id': 'ad3fcf9d-7c96-4db9-adf4-3c5e4852540e'}]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[i][\"annotations\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'end': 20662, 'tag': 'USERNAME', 'start': 20654, 'value': 'rheineke', 'tag_id': 'c1f50281-3cff-4eee-85d4-212d99963238', 'correct': True, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-08T14:47:37.582428+00:00', 'annotator_id': 4}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': 'e0ad7e43-6233-4346-98b0-f72f9b6b35ea'}, {'end': 55, 'tag': 'EMAIL', 'start': 32, 'value': 'reece.heineke@gmail.com', 'tag_id': '76b4fbc7-b129-40ea-a6d3-8988fe626973', 'correct': True, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'loubnabenallal1999@gmail.com', 'timestamp': '2022-11-05T17:58:17.881+00:00', 'annotator_id': 1}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': 'ad00e862-d92a-4cc0-92ea-a0e97b70f5f7'}, {'end': 31, 'tag': 'NAME', 'start': 18, 'value': 'Reece Heineke', 'tag_id': '555cc074-289d-40b7-8e11-6221bb7c7167', 'correct': True, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-07T09:38:24.735572+00:00', 'annotator_id': 4}, {'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-08T15:12:14.564192+00:00', 'annotator_id': 4}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': '8c272f1a-554e-48ce-8bd5-8d96ea496293'}]\n",
      "\n",
      "\n",
      "[{'end': 20662, 'tag': 'AMBIGUOUS', 'start': 20654, 'value': 'rheineke', 'tag_id': '3687481c-bf1c-423e-bf1d-287d6a5c67e0', 'correct': False, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-08T14:47:30.185752+00:00', 'annotator_id': 4}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': '8dd1106f-75eb-4184-96d1-865f0e516e13'}, {'end': 30, 'tag': 'AMBIGUOUS', 'start': 17, 'value': ' Reece Heinek', 'tag_id': '3687481c-bf1c-423e-bf1d-287d6a5c67e0', 'correct': False, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-07T09:34:55.441427+00:00', 'annotator_id': 4}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': '5028a874-0c96-459d-9887-2ba6ca67dfc1'}, {'end': 31, 'tag': 'AMBIGUOUS', 'start': 18, 'value': 'Reece Heineke', 'tag_id': '3687481c-bf1c-423e-bf1d-287d6a5c67e0', 'correct': False, 'reviewed': True, 'example_id': '00f9b923-ab9d-4373-a6e0-d6172f7999fd', 'annotated_by': [{'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-08T15:12:11.995254+00:00', 'annotator_id': 4}, {'annotator': 'christopher.akiki@gmail.com', 'timestamp': '2022-11-07T09:38:19.96111+00:00', 'annotator_id': 4}], 'definition_id': 'a6e0c132-cd58-45d1-8dd5-5af8cdc6d399', 'tagged_token_id': 'b1c1fcf5-529e-4503-a637-b6f74e47bf4c'}]\n"
     ]
    }
   ],
   "source": [
    "# example annotations: those rejected by reviewers are kept with the tag correct: False, let's remove them\n",
    "example = dataset[362]\n",
    "print([a for a in example[\"annotations\"] if a[\"correct\"]])\n",
    "print(\"\\n\")\n",
    "print([a for a in example[\"annotations\"] if not a[\"correct\"]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kept 1016 annotations, rejected 302 annotations\n"
     ]
    }
   ],
   "source": [
    "# remove annotations where \"correct\" is False\n",
    "rejected = 0\n",
    "kept = 0\n",
    "list_rejections = []\n",
    "for i, example in enumerate(dataset):\n",
    "    # only keep correct annotations (after review)\n",
    "    correct_annotations = [a for a in example[\"annotations\"] if a[\"correct\"]]\n",
    "    list_rejections += [a for a in example['annotations'] if not a['correct']]\n",
    "    #update dataset\n",
    "    dataset[i][\"annotations\"] = correct_annotations\n",
    "    \n",
    "print(f\"kept {sum([len(ex['annotations']) for ex in dataset])} annotations, rejected {len(list_rejections)} annotations\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kept 1016 annotations, rejected 302 annotations\n"
     ]
    }
   ],
   "source": [
    "# remove annotations where \"correct\" is False\n",
    "rejected = 0\n",
    "kept = 0\n",
    "list_rejections = []\n",
    "for i, example in enumerate(dataset):\n",
    "    # only keep correct annotations (after review)\n",
    "    correct_annotations = [a for a in example[\"annotations\"] if a[\"correct\"]]\n",
    "    list_rejections += [a for a in example['annotations'] if not a['correct']]\n",
    "    #update dataset\n",
    "    dataset[i][\"annotations\"] = correct_annotations\n",
    "    \n",
    "print(f\"kept {sum([len(ex['annotations']) for ex in dataset])} annotations, rejected {len(list_rejections)} annotations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the dataset for `datasets` library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import datasets\n",
    "import pandas as pd\n",
    "from copy import deepcopy\n",
    "\n",
    "def convert_key(example):\n",
    "    sample = deepcopy(example)\n",
    "    if sample[\"tag\"] in [\"API_KEY\", \"SSH_KEY\"]:\n",
    "        sample[\"tag\"] = \"KEY\"\n",
    "    return sample\n",
    "\n",
    "def process_example(example):\n",
    "    new_sample = {}\n",
    "    new_sample[\"content\"] = example[\"content\"]\n",
    "    new_sample[\"language\"] = example[\"metadata\"][\"lang\"]\n",
    "    new_sample[\"license\"] = example[\"metadata\"][\"licenses\"][0]\n",
    "    new_sample[\"path\"] = example[\"metadata\"][\"repository_name\"] + \"/\" + example[\"metadata\"][\"path\"]\n",
    "    new_sample[\"annotation_id\"] = example[\"example_id\"]\n",
    "    new_sample[\"pii\"] = []\n",
    "    new_sample[\"pii_modified\"] = []\n",
    "    for annotation in example[\"annotations\"]:\n",
    "        pii = {\"tag\": annotation[\"tag\"],\n",
    "                \"value\": annotation[\"value\"],\n",
    "                \"start\": annotation[\"start\"],\n",
    "                \"end\": annotation[\"end\"]}\n",
    "        start = max(0, pii[\"start\"] - 50)\n",
    "        end = min(len(new_sample[\"content\"]), pii[\"end\"] + 50)\n",
    "        pii[\"context\"] = new_sample[\"content\"][start:end]\n",
    "        # column with one Key tag for both API Keys and SSH keys\n",
    "        modified_pii = convert_key(pii)\n",
    "        new_sample[\"pii\"].append(pii)\n",
    "        new_sample[\"pii_modified\"].append(modified_pii)\n",
    "    # we save the pii in json to avoid not matching sizes per column in `datasets`\n",
    "    new_sample[\"pii\"] = json.dumps(new_sample[\"pii\"])\n",
    "    new_sample[\"pii_modified\"] = json.dumps(new_sample[\"pii_modified\"])\n",
    "    return new_sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'tag': 'API_KEY',\n",
       "  'value': '476611152863-ltgqfk9jhq1vsenin5039n58ogkraltb.apps.googleusercontent.com',\n",
       "  'start': 6842,\n",
       "  'end': 6914,\n",
       "  'context': 'onScheme;\\n\\n                   options.ClientId = \"476611152863-ltgqfk9jhq1vsenin5039n58ogkraltb.apps.googleusercontent.com\";\\n                   options.ClientSecret = \"rSHv'},\n",
       " {'tag': 'API_KEY',\n",
       "  'value': '99eb0b9d-ca40-476e-b5ac-6f4c32bfb530',\n",
       "  'start': 7336,\n",
       "  'end': 7372,\n",
       "  'context': ' false };\\n                    options.ClientId = \"99eb0b9d-ca40-476e-b5ac-6f4c32bfb530\";\\n                    options.CallbackPath = \"/si'},\n",
       " {'tag': 'USERNAME',\n",
       "  'value': 'aspnet',\n",
       "  'start': 7783,\n",
       "  'end': 7789,\n",
       "  'context': '         // And\\n            // https://github.com/aspnet/Docs/issues/2384#issuecomment-297980490\\n         '},\n",
       " {'tag': 'USERNAME',\n",
       "  'value': 'openiddict',\n",
       "  'start': 7692,\n",
       "  'end': 7702,\n",
       "  'context': ' env)\\n        {\\n            // https://github.com/openiddict/openiddict-core/issues/518\\n            // And\\n   '},\n",
       " {'tag': 'API_KEY',\n",
       "  'value': 'rSHvhgdOQUB4KMc5JS1alzhg',\n",
       "  'start': 6960,\n",
       "  'end': 6984,\n",
       "  'context': '.com\";\\n                   options.ClientSecret = \"rSHvhgdOQUB4KMc5JS1alzhg\";\\n               })\\n               .AddOpenIdConn'}]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# small test\n",
    "res = process_example(dataset[360])\n",
    "print(res.keys())\n",
    "pii = json.loads(res[\"pii\"])\n",
    "pii"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'tag': 'KEY',\n",
       "  'value': '476611152863-ltgqfk9jhq1vsenin5039n58ogkraltb.apps.googleusercontent.com',\n",
       "  'start': 6842,\n",
       "  'end': 6914,\n",
       "  'context': 'onScheme;\\n\\n                   options.ClientId = \"476611152863-ltgqfk9jhq1vsenin5039n58ogkraltb.apps.googleusercontent.com\";\\n                   options.ClientSecret = \"rSHv'},\n",
       " {'tag': 'KEY',\n",
       "  'value': '99eb0b9d-ca40-476e-b5ac-6f4c32bfb530',\n",
       "  'start': 7336,\n",
       "  'end': 7372,\n",
       "  'context': ' false };\\n                    options.ClientId = \"99eb0b9d-ca40-476e-b5ac-6f4c32bfb530\";\\n                    options.CallbackPath = \"/si'},\n",
       " {'tag': 'USERNAME',\n",
       "  'value': 'aspnet',\n",
       "  'start': 7783,\n",
       "  'end': 7789,\n",
       "  'context': '         // And\\n            // https://github.com/aspnet/Docs/issues/2384#issuecomment-297980490\\n         '},\n",
       " {'tag': 'USERNAME',\n",
       "  'value': 'openiddict',\n",
       "  'start': 7692,\n",
       "  'end': 7702,\n",
       "  'context': ' env)\\n        {\\n            // https://github.com/openiddict/openiddict-core/issues/518\\n            // And\\n   '},\n",
       " {'tag': 'KEY',\n",
       "  'value': 'rSHvhgdOQUB4KMc5JS1alzhg',\n",
       "  'start': 6960,\n",
       "  'end': 6984,\n",
       "  'context': '.com\";\\n                   options.ClientSecret = \"rSHvhgdOQUB4KMc5JS1alzhg\";\\n               })\\n               .AddOpenIdConn'}]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pii = json.loads(res[\"pii_modified\"])\n",
    "pii"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build a dataset\n",
    "def build_dataset(data):\n",
    "    df = pd.DataFrame([process_example(ex) for ex in data])\n",
    "    dataset = datasets.Dataset.from_pandas(df)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 549,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf_dataset = build_dataset(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61b22d4ed13d496d9bec51876d270837",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3861956"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_dataset.to_json(\"prefiltered_v2.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manual reviewing of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i += 1\n",
    "pii = json.loads(hf_dataset[i][\"pii\"])\n",
    "for annot in pii:\n",
    "    print(i, annot[\"tag\"], annot[\"value\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* example 34 has a wrong password, example 158 has a wrong IP address, 206 has a wrong passwor and 241 has many API keys that don't look like tokens and are propably passwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 550,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add id column to dataset as a new column\n",
    "hf_dataset = hf_dataset.add_column(\"id\", [i for i in range(len(hf_dataset))])\n",
    "hf_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_pii(example):\n",
    "    new_example = deepcopy(example)\n",
    "    if example[\"id\"] in [34, 206]:\n",
    "        pii = json.loads(example[\"pii\"])\n",
    "        # remove wrong password\n",
    "        new_example[\"pii\"] = json.dumps(pii[1:])\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii[1:])\n",
    "\n",
    "    elif example[\"id\"] == 158:\n",
    "        pii = json.loads(hf_dataset[158][\"pii\"])\n",
    "        # remove wrong IP_ADDRESS\n",
    "        pii = pii[:3] + pii[4:]\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "\n",
    "    elif example[\"id\"] == 241:\n",
    "        pii = json.loads(example[\"pii\"])\n",
    "        for id, e in enumerate(pii):\n",
    "            if e[\"value\"] == \"AIzaasdf\":\n",
    "                pii[id][\"tag\"] = \"PASSWORD\"\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "    return new_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 552,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e2cc34b78afa480c8e8149150e00913c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/400 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hf_dataset_2 = hf_dataset.map(update_pii, features=hf_dataset.features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 553,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf_dataset_2 = hf_dataset_2.remove_columns([\"id\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 558,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 558,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_dataset_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 557,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34 PASSWORD wrongpwd\n"
     ]
    }
   ],
   "source": [
    "id = 34\n",
    "pii = json.loads(hf_dataset_2[id][\"pii\"])\n",
    "for annot in pii:\n",
    "    print(id, annot[\"tag\"], annot[\"value\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove file at index 11 many ambigous keys\n",
    "# remove file 51 many incorrect names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 569,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'indices'=<generator object <genexpr> at 0x7fd6c1dd8430> of the transform datasets.arrow_dataset.Dataset.select couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
     ]
    }
   ],
   "source": [
    "hf_dataset_3 = hf_dataset_2.select((i for i in range(len(hf_dataset_2)) if i!=11 and i!=51))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# samples kept: 224\n"
     ]
    }
   ],
   "source": [
    "with open('data_lightag/pii-first-final.json') as f:\n",
    "    labels_no_filter = json.load(f)\n",
    "\n",
    "def build_dataset_n(labels):\n",
    "    import pandas as pd\n",
    "    examples = labels[\"examples\"]\n",
    "    df = pd.DataFrame([process_example(ex) for ex in examples])\n",
    "    dataset = datasets.Dataset.from_pandas(df)\n",
    "    return dataset\n",
    "\n",
    "# remove examples where seen_by is empty\n",
    "L = []\n",
    "for example in labels_no_filter[\"examples\"]:\n",
    "    if example[\"seen_by\"]:\n",
    "        L.append(example)\n",
    "labels_no_filter[\"examples\"] = L\n",
    "print(f\"# samples kept: {len(labels_no_filter['examples'])}\")\n",
    "\n",
    "dataset_no_filter = build_dataset_n(labels_no_filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 568,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'],\n",
       "    num_rows: 2\n",
       "})"
      ]
     },
     "execution_count": 568,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "two_samples = dataset_no_filter.select([223, 218])\n",
    "two_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "ds_clean = datasets.concatenate_datasets([hf_dataset_3, two_samples])\n",
    "ds_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 573,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e6f1c33ee2bb48c28eb368dcbfbe6eaa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3724333"
      ]
     },
     "execution_count": 573,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_clean.to_json(\"prefiltered_final.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 574,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c14a9610ffc4f528b3fd393a3e2aea8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Pushing dataset shards to the dataset hub:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ds_clean.push_to_hub(\"dummy_data_clean\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More cleaning after we ran detect-secrets and spotted mis annotations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration bigcode--pii-for-code-72a7cff2b59e4251\n",
      "Found cached dataset json (/Users/loubnabenallal/.cache/huggingface/datasets/bigcode___json/bigcode--pii-for-code-72a7cff2b59e4251/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"bigcode/pii-for-code\", use_auth_token=True, split=\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pii []\n",
      "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.m1U63blB0MLej_WfB7yC2FTMnCziif9X8yzwDEfJXAg\n",
      " *      responses:\n",
      " *        \"204\":\n",
      " *          description: No content\n",
      " *        \"404\":\n",
      " *          $ref: '#/components/responses/NotFound'\n",
      " */\n",
      "\n",
      "/**\n",
      " * @swagger\n",
      " * paths:\n",
      " *  /auth/refresh-tokens:\n",
      " *    post:\n",
      " *      summary: Refresh auth tokens\n",
      " *      tags: [Auth]\n",
      " *      requestBody:\n",
      " *        required: true\n",
      " *        content:\n",
      " *          application/json:\n",
      " *            schema:\n",
      " *              type: object\n",
      " *              required:\n",
      " *                - refreshToken\n",
      " *              properties:\n",
      " *                refreshToken:\n",
      " *                  type: string\n",
      " *              example:\n",
      " *                refreshToken: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.m1U63blB0MLej_WfB7yC2FTMnCziif9X8yzwDEfJXAg\n",
      " *      responses:\n",
      " *        \"200\":\n",
      " *          description: OK\n",
      " *          content:\n",
      " *            application/json:\n",
      " *              schema:\n",
      " *                $ref: '#/components/schemas/AuthTokens'\n",
      " *        \"401\":\n",
      " *          $ref: '#/components/responses/Unauthorized'\n",
      " */\n",
      "\n",
      "/**\n",
      " * @swagger\n",
      " * paths:\n",
      " *  /auth/forgot-password:\n",
      " *    post:\n",
      " *      summary: Forgot password\n",
      " *      description: An email will be sent to reset password.\n",
      " *      tags: [Auth]\n",
      " *      requestBody:\n",
      " *        required: true\n",
      " *        content:\n",
      " *          application/json:\n",
      " *            schema:\n",
      " *              type: object\n",
      " *              required:\n",
      " *                - email\n",
      " *              properties:\n",
      " *                email:\n",
      " *                  type: string\n",
      " *                  format: email\n",
      " *              example:\n",
      " *                email: fake@example.com\n",
      " *      responses:\n",
      " *        \"204\":\n",
      " *          description: No content\n",
      " *        \"404\":\n",
      " *          $ref: '#/components/responses/NotFound'\n",
      " */\n",
      "\n",
      "/**\n",
      " * @swagger\n",
      " * paths:\n",
      " *  /auth/reset-password:\n",
      " *    post:\n",
      " *      summary: Reset password\n",
      " *      tags: [Auth]\n",
      " *      parameters:\n",
      " *        - in: query\n",
      " *          name: token\n",
      " *          required: true\n",
      " *          schema:\n",
      " *            type: string\n",
      " *          description: The reset password token\n",
      " *      requestBody:\n",
      " *        required: true\n",
      " *        content:\n",
      " *          application/json:\n",
      " *            schema:\n",
      " *              type: object\n",
      " *              required:\n",
      " *                - password\n",
      " *              properties:\n",
      " *                password:\n",
      " *                  type: string\n",
      " *                  format: password\n",
      " *                  minLength: 8\n",
      " *                  description: At least one number and one letter\n",
      " *              example:\n",
      " *                password: password1\n",
      " *      responses:\n",
      " *        \"204\":\n",
      " *          description: No content\n",
      " *        \"401\":\n",
      " *          description: Password reset failed\n",
      " *          content:\n",
      " *            application/json:\n",
      " *              schema:\n",
      " *                $ref: '#/components/schemas/Error'\n",
      " *              example:\n",
      " *                code: 401\n",
      " *                message: Password reset failed\n",
      " */\n",
      "\n"
     ]
    }
   ],
   "source": [
    "i = 317\n",
    "pii = json.loads(ds[i][\"pii\"])\n",
    "print(\"pii\", pii)\n",
    "print(ds[i][\"content\"][4098:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pii [{'tag': 'NAME', 'value': 'Satoshi Nakamoto', 'start': 22, 'end': 38, 'context': '// Copyright (c) 2010 Satoshi Nakamoto\\n// Original Code: Copyright (c) 2009-2014 The Bit'}]\n",
      "12tvKAXCxZjSmdNbao16dKXC8tRWfcF5oc\\\"   (string) bitmark address\\n\"\n",
      "            \"           ,...\\n\"\n",
      "            \"         ]\\n\"\n",
      "            \"       }\\n\"\n",
      "            \"     }\\n\"\n",
      "            \"     ,...\\n\"\n",
      "            \"  ],\\n\"\n",
      "            \"}\\n\"\n",
      "\n",
      "            \"\\nExamples:\\n\"\n",
      "            + HelpExampleCli(\"decoderawtransaction\", \"\\\"hexstring\\\"\")\n",
      "            + HelpExampleRpc(\"decoderawtransaction\", \"\\\"hexstring\\\"\")\n",
      "        );\n",
      "\n",
      "    vector<unsigned char> txData(ParseHexV(params[0], \"argument\"));\n",
      "    CDataStream ssData(txData, SER_NETWORK, PROTOCOL_VERSION);\n",
      "    CTransaction tx;\n",
      "    try {\n",
      "        ssData >> tx;\n",
      "    }\n",
      "    catch (std::exception &e) {\n",
      "        throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"TX decode failed\");\n",
      "    }\n",
      "\n",
      "    Object result;\n",
      "    TxToJSON(tx, 0, result);\n",
      "\n",
      "    return result;\n",
      "}\n",
      "\n",
      "Value decodescript(const Array& params, bool fHelp)\n",
      "{\n",
      "    if (fHelp || params.size() != 1)\n",
      "        throw runtime_error(\n",
      "            \"decodescript \\\"hex\\\"\\n\"\n",
      "            \"\\nDecode a hex-encoded script.\\n\"\n",
      "            \"\\nArguments:\\n\"\n",
      "            \"1. \\\"hex\\\"     (string) the hex encoded script\\n\"\n",
      "            \"\\nResult:\\n\"\n",
      "            \"{\\n\"\n",
      "            \"  \\\"asm\\\":\\\"asm\\\",   (string) Script public key\\n\"\n",
      "            \"  \\\"hex\\\":\\\"hex\\\",   (string) hex encoded public key\\n\"\n",
      "            \"  \\\"type\\\":\\\"type\\\", (string) The output type\\n\"\n",
      "            \"  \\\"reqSigs\\\": n,    (numeric) The required signatures\\n\"\n",
      "            \"  \\\"addresses\\\": [   (json array of string)\\n\"\n",
      "            \"     \\\"address\\\"     (string) bitmark address\\n\"\n",
      "            \"     ,...\\n\"\n",
      "            \"  ],\\n\"\n",
      "            \"  \\\"p2sh\\\",\\\"address\\\" (string) script address\\n\"\n",
      "            \"}\\n\"\n",
      "            \"\\nExamples:\\n\"\n",
      "            + HelpExampleCli(\"decodescript\", \"\\\"hexstring\\\"\")\n",
      "            + HelpExampleRpc(\"decodescript\", \"\\\"hexstring\\\"\")\n",
      "        );\n",
      "\n",
      "    RPCTypeCheck(params, list_of(str_type));\n",
      "\n",
      "    Object r;\n",
      "    CScript script;\n",
      "    if (params[0].get_str().size() > 0){\n",
      "        vector<unsigned char> scriptData(ParseHexV(params[0], \"argument\"));\n",
      "        script = CScript(scriptData.begin(), scriptData.end());\n",
      "    } else {\n",
      "        // Empty scripts are valid\n",
      "    }\n",
      "    ScriptPubKeyToJSON(script, r, false);\n",
      "\n",
      "    r.push_back(Pair(\"p2sh\", CBitmarkAddress(script.GetID()).ToString()));\n",
      "    return r;\n",
      "}\n",
      "\n",
      "Value signrawtransaction(const Array& params, bool fHelp)\n",
      "{\n",
      "    if (fHelp || params.size() < 1 || params.size() > 4)\n",
      "        throw runtime_error(\n",
      "            \"signrawtransaction \\\"hexstring\\\" ( [{\\\"txid\\\":\\\"id\\\",\\\"vout\\\":n,\\\"scriptPubKey\\\":\\\"hex\\\",\\\"redeemScript\\\":\\\"hex\\\"},...] [\\\"privatekey1\\\",...] sighashtype )\\n\"\n",
      "            \"\\nSign inputs for raw transaction (serialized, hex-encoded).\\n\"\n",
      "            \"The second optional argument (may be null) is an array of previous transaction outputs that\\n\"\n",
      "            \"this transaction depends on but may not yet be in the block chain.\\n\"\n",
      "            \"The third optional argument (may be null) is an array of base58-encoded private\\n\"\n",
      "            \"keys that, if given, will be the only keys used to sign the transaction.\\n\"\n",
      "#ifdef ENABLE_WALLET\n",
      "            + HelpRequiringPassphrase() + \"\\n\"\n",
      "#endif\n",
      "\n",
      "            \"\\nArguments:\\n\"\n",
      "            \"1. \\\"hexstring\\\"     (string, required) The transaction hex string\\n\"\n",
      "            \"2. \\\"prevtxs\\\"       (string, optional) An json array of previous dependent transaction outputs\\n\"\n",
      "            \"     [               (json array of json objects, or 'null' if none provided)\\n\"\n",
      "            \"       {\\n\"\n",
      "            \"         \\\"txid\\\":\\\"id\\\",             (string, required) The transaction id\\n\"\n",
      "            \"         \\\"vout\\\":n,                  (numeric, required) The output number\\n\"\n",
      "            \"         \\\"scriptPubKey\\\": \\\"hex\\\",   (string, required) script key\\n\"\n",
      "            \"         \\\"redeemScript\\\": \\\"hex\\\"    (string, required for P2SH) redeem script\\n\"\n",
      "            \"       }\\n\"\n",
      "            \"       ,...\\n\"\n",
      "            \"    ]\\n\"\n",
      "            \"3. \\\"privatekeys\\\"     (string, optional) A json array of base58-encoded private keys for signing\\n\"\n",
      "            \"    [                  (json array of strings, or 'null' if none provided)\\n\"\n",
      "            \"      \\\"privatekey\\\"   (string) private key in base58-encoding\\n\"\n",
      "            \"      ,...\\n\"\n",
      "            \"    ]\\n\"\n",
      "            \"4. \\\"sighashtype\\\"     (string, optional, default=ALL) The signature hash type. Must be one of\\n\"\n",
      "            \"       \\\"ALL\\\"\\n\"\n",
      "            \"       \\\"NONE\\\"\\n\"\n",
      "            \"       \\\"SINGLE\\\"\\n\"\n",
      "            \"       \\\"ALL|ANYONECANPAY\\\"\\n\"\n",
      "            \"       \\\"NONE|ANYONECANPAY\\\"\\n\"\n",
      "            \"       \\\"SINGLE|ANYONECANPAY\\\"\\n\"\n",
      "\n",
      "            \"\\nResult:\\n\"\n",
      "            \"{\\n\"\n",
      "            \"  \\\"hex\\\": \\\"value\\\",   (string) The raw transaction with signature(s) (hex-encoded string)\\n\"\n",
      "            \"  \\\"complete\\\": n       (numeric) if transaction has a complete set of signature (0 if not)\\n\"\n",
      "            \"}\\n\"\n",
      "\n",
      "            \"\\nExamples:\\n\"\n",
      "            + HelpExampleCli(\"signrawtransaction\", \"\\\"myhex\\\"\")\n",
      "            + HelpExampleRpc(\"signrawtransaction\", \"\\\"myhex\\\"\")\n",
      "        );\n",
      "\n",
      "    RPCTypeCheck(params, list_of(str_type)(array_type)(array_type)(str_type), true);\n",
      "\n",
      "    vector<unsigned char> txData(ParseHexV(params[0], \"argument 1\"));\n",
      "    CDataStream ssData(txData, SER_NETWORK, PROTOCOL_VERSION);\n",
      "    vector<CTransaction> txVariants;\n",
      "    while (!ssData.empty())\n",
      "    {\n",
      "        try {\n",
      "            CTransaction tx;\n",
      "            ssData >> tx;\n",
      "            txVariants.push_back(tx);\n",
      "        }\n",
      "        catch (std::exception &e) {\n",
      "            throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"TX decode failed\");\n",
      "        }\n",
      "    }\n",
      "\n",
      "    if (txVariants.empty())\n",
      "        throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"Missing transaction\");\n",
      "\n",
      "    // mergedTx will end up with all the signatures; it\n",
      "    // starts as a clone of the rawtx:\n",
      "    CTransaction mergedTx(txVariants[0]);\n",
      "    bool fComplete = true;\n",
      "\n",
      "    // Fetch previous transactions (inputs):\n",
      "    CCoinsView viewDummy;\n",
      "    CCoinsViewCache view(viewDummy);\n",
      "    {\n",
      "        LOCK(mempool.cs);\n",
      "        CCoinsViewCache &viewChain = *pcoinsTip;\n",
      "        CCoinsViewMemPool viewMempool(viewChain, mempool);\n",
      "        view.SetBackend(viewMempool); // temporarily switch cache backend to db+mempool view\n",
      "\n",
      "        BOOST_FOREACH(const CTxIn& txin, mergedTx.vin) {\n",
      "            const uint256& prevHash = txin.prevout.hash;\n",
      "            CCoins coins;\n",
      "            view.GetCoins(prevHash, coins); // this is certainly allowed to fail\n",
      "        }\n",
      "\n",
      "        view.SetBackend(viewDummy); // switch back to avoid locking mempool for too long\n",
      "    }\n",
      "\n",
      "    bool fGivenKeys = false;\n",
      "    CBasicKeyStore tempKeystore;\n",
      "    if (params.size() > 2 && params[2].type() != null_type)\n",
      "    {\n",
      "        fGivenKeys = true;\n",
      "        Array keys = params[2].get_array();\n",
      "        BOOST_FOREACH(Value k, keys)\n",
      "        {\n",
      "            CBitmarkSecret vchSecret;\n",
      "            bool fGood = vchSecret.SetString(k.get_str());\n",
      "            if (!fGood)\n",
      "                throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, \"Invalid private key\");\n",
      "            CKey key = vchSecret.GetKey();\n",
      "            tempKeystore.AddKey(key);\n",
      "        }\n",
      "    }\n",
      "#ifdef ENABLE_WALLET\n",
      "    else\n",
      "        EnsureWalletIsUnlocked();\n",
      "#endif\n",
      "\n",
      "    // Add previous txouts given in the RPC call:\n",
      "    if (params.size() > 1 && params[1].type() != null_type)\n",
      "    {\n",
      "        Array prevTxs = params[1].get_array();\n",
      "        BOOST_FOREACH(Value& p, prevTxs)\n",
      "        {\n",
      "            if (p.type() != obj_type)\n",
      "                throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"expected object with {\\\"txid'\\\",\\\"vout\\\",\\\"scriptPubKey\\\"}\");\n",
      "\n",
      "            Object prevOut = p.get_obj();\n",
      "\n",
      "            RPCTypeCheck(prevOut, map_list_of(\"txid\", str_type)(\"vout\", int_type)(\"scriptPubKey\", str_type));\n",
      "\n",
      "            uint256 txid = ParseHashO(prevOut, \"txid\");\n",
      "\n",
      "            int nOut = find_value(prevOut, \"vout\").get_int();\n",
      "            if (nOut < 0)\n",
      "                throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"vout must be positive\");\n",
      "\n",
      "            vector<unsigned char> pkData(ParseHexO(prevOut, \"scriptPubKey\"));\n",
      "            CScript scriptPubKey(pkData.begin(), pkData.end());\n",
      "\n",
      "            CCoins coins;\n",
      "            if (view.GetCoins(txid, coins)) {\n",
      "                if (coins.IsAvailable(nOut) && coins.vout[nOut].scriptPubKey != scriptPubKey) {\n",
      "                    string err(\"Previous output scriptPubKey mismatch:\\n\");\n",
      "                    err = err + coins.vout[nOut].scriptPubKey.ToString() + \"\\nvs:\\n\"+\n",
      "                        scriptPubKey.ToString();\n",
      "                    throw JSONRPCError(RPC_DESERIALIZATION_ERROR, err);\n",
      "                }\n",
      "                // what todo if txid is known, but the actual output isn't?\n",
      "            }\n",
      "            if ((unsigned int)nOut >= coins.vout.size())\n",
      "                coins.vout.resize(nOut+1);\n",
      "            coins.vout[nOut].scriptPubKey = scriptPubKey;\n",
      "            coins.vout[nOut].nValue = 0; // we don't know the actual output value\n",
      "            view.SetCoins(txid, coins);\n",
      "\n",
      "            // if redeemScript given and not using the local wallet (private keys\n",
      "            // given), add redeemScript to the tempKeystore so it can be signed:\n",
      "            if (fGivenKeys && scriptPubKey.IsPayToScriptHash())\n",
      "            {\n",
      "                RPCTypeCheck(prevOut, map_list_of(\"txid\", str_type)(\"vout\", int_type)(\"scriptPubKey\", str_type)(\"redeemScript\",str_type));\n",
      "                Value v = find_value(prevOut, \"redeemScript\");\n",
      "                if (!(v == Value::null))\n",
      "                {\n",
      "                    vector<unsigned char> rsData(ParseHexV(v, \"redeemScript\"));\n",
      "                    CScript redeemScript(rsData.begin(), rsData.end());\n",
      "                    tempKeystore.AddCScript(redeemScript);\n",
      "                }\n",
      "            }\n",
      "        }\n",
      "    }\n",
      "\n",
      "#ifdef ENABLE_WALLET\n",
      "    const CKeyStore& keystore = ((fGivenKeys || !pwalletMain) ? tempKeystore : *pwalletMain);\n",
      "#else\n",
      "    const CKeyStore& keystore = tempKeystore;\n",
      "#endif\n",
      "\n",
      "    int nHashType = SIGHASH_ALL;\n",
      "    if (params.size() > 3 && params[3].type() != null_type)\n",
      "    {\n",
      "        static map<string, int> mapSigHashValues =\n",
      "            boost::assign::map_list_of\n",
      "            (string(\"ALL\"), int(SIGHASH_ALL))\n",
      "            (string(\"ALL|ANYONECANPAY\"), int(SIGHASH_ALL|SIGHASH_ANYONECANPAY))\n",
      "            (string(\"NONE\"), int(SIGHASH_NONE))\n",
      "            (string(\"NONE|ANYONECANPAY\"), int(SIGHASH_NONE|SIGHASH_ANYONECANPAY))\n",
      "            (string(\"SINGLE\"), int(SIGHASH_SINGLE))\n",
      "            (string(\"SINGLE|ANYONECANPAY\"), int(SIGHASH_SINGLE|SIGHASH_ANYONECANPAY))\n",
      "            ;\n",
      "        string strHashType = params[3].get_str();\n",
      "        if (mapSigHashValues.count(strHashType))\n",
      "            nHashType = mapSigHashValues[strHashType];\n",
      "        else\n",
      "            throw JSONRPCError(RPC_INVALID_PARAMETER, \"Invalid sighash param\");\n",
      "    }\n",
      "\n",
      "    bool fHashSingle = ((nHashType & ~SIGHASH_ANYONECANPAY) == SIGHASH_SINGLE);\n",
      "\n",
      "    // Sign what we can:\n",
      "    for (unsigned int i = 0; i < mergedTx.vin.size(); i++)\n",
      "    {\n",
      "        CTxIn& txin = mergedTx.vin[i];\n",
      "        CCoins coins;\n",
      "        if (!view.GetCoins(txin.prevout.hash, coins) || !coins.IsAvailable(txin.prevout.n))\n",
      "        {\n",
      "            fComplete = false;\n",
      "            continue;\n",
      "        }\n",
      "        const CScript& prevPubKey = coins.vout[txin.prevout.n].scriptPubKey;\n",
      "\n",
      "        txin.scriptSig.clear();\n",
      "        // Only sign SIGHASH_SINGLE if there's a corresponding output:\n",
      "        if (!fHashSingle || (i < mergedTx.vout.size()))\n",
      "            SignSignature(keystore, prevPubKey, mergedTx, i, nHashType);\n",
      "\n",
      "        // ... and merge in other signatures:\n",
      "        BOOST_FOREACH(const CTransaction& txv, txVariants)\n",
      "        {\n",
      "            txin.scriptSig = CombineSignatures(prevPubKey, mergedTx, i, txin.scriptSig, txv.vin[i].scriptSig);\n",
      "        }\n",
      "        if (!VerifyScript(txin.scriptSig, prevPubKey, mergedTx, i, SCRIPT_VERIFY_P2SH | SCRIPT_VERIFY_STRICTENC | SCRIPT_VERIFY_CHECKLOCKTIMEVERIFY, 0))\n",
      "            fComplete = false;\n",
      "    }\n",
      "\n",
      "    Object result;\n",
      "    CDataStream ssTx(SER_NETWORK, PROTOCOL_VERSION);\n",
      "    ssTx << mergedTx;\n",
      "    result.push_back(Pair(\"hex\", HexStr(ssTx.begin(), ssTx.end())));\n",
      "    result.push_back(Pair(\"complete\", fComplete));\n",
      "\n",
      "    return result;\n",
      "}\n",
      "\n",
      "Value sendrawtransaction(const Array& params, bool fHelp)\n",
      "{\n",
      "    if (fHelp || params.size() < 1 || params.size() > 2)\n",
      "        throw runtime_error(\n",
      "            \"sendrawtransaction \\\"hexstring\\\" ( allowhighfees )\\n\"\n",
      "            \"\\nSubmits raw transaction (serialized, hex-encoded) to local node and network.\\n\"\n",
      "            \"\\nAlso see createrawtransaction and signrawtransaction calls.\\n\"\n",
      "            \"\\nArguments:\\n\"\n",
      "            \"1. \\\"hexstring\\\"    (string, required) The hex string of the raw transaction)\\n\"\n",
      "            \"2. allowhighfees    (boolean, optional, default=false) Allow high fees\\n\"\n",
      "            \"\\nResult:\\n\"\n",
      "            \"\\\"hex\\\"             (string) The transaction hash in hex\\n\"\n",
      "            \"\\nExamples:\\n\"\n",
      "            \"\\nCreate a transaction\\n\"\n",
      "            + HelpExampleCli(\"createrawtransaction\", \"\\\"[{\\\\\\\"txid\\\\\\\" : \\\\\\\"mytxid\\\\\\\",\\\\\\\"vout\\\\\\\":0}]\\\" \\\"{\\\\\\\"myaddress\\\\\\\":0.01}\\\"\") +\n",
      "            \"Sign the transaction, and get back the hex\\n\"\n",
      "            + HelpExampleCli(\"signrawtransaction\", \"\\\"myhex\\\"\") +\n",
      "            \"\\nSend the transaction (signed hex)\\n\"\n",
      "            + HelpExampleCli(\"sendrawtransaction\", \"\\\"signedhex\\\"\") +\n",
      "            \"\\nAs a json rpc call\\n\"\n",
      "            + HelpExampleRpc(\"sendrawtransaction\", \"\\\"signedhex\\\"\")\n",
      "        );\n",
      "\n",
      "\n",
      "    // parse hex string from parameter\n",
      "    vector<unsigned char> txData(ParseHexV(params[0], \"parameter\"));\n",
      "    CDataStream ssData(txData, SER_NETWORK, PROTOCOL_VERSION);\n",
      "    CTransaction tx;\n",
      "\n",
      "    bool fOverrideFees = false;\n",
      "    if (params.size() > 1)\n",
      "        fOverrideFees = params[1].get_bool();\n",
      "\n",
      "    // deserialize binary data stream\n",
      "    try {\n",
      "        ssData >> tx;\n",
      "    }\n",
      "    catch (std::exception &e) {\n",
      "        throw JSONRPCError(RPC_DESERIALIZATION_ERROR, \"TX decode failed\");\n",
      "    }\n",
      "    uint256 hashTx = tx.GetHash();\n",
      "\n",
      "    CCoinsViewCache &view = *pcoinsTip;\n",
      "    CCoins existingCoins;\n",
      "    bool fHaveMempool = mempool.exists(hashTx);\n",
      "    bool fHaveChain = view.GetCoins(hashTx, existingCoins) && existingCoins.nHeight < 1000000000;\n",
      "    if (!fHaveMempool && !fHaveChain) {\n",
      "        // push to local node and sync with wallets\n",
      "        CValidationState state;\n",
      "        if (AcceptToMemoryPool(mempool, state, tx, false, NULL, !fOverrideFees))\n",
      "            SyncWithWallets(hashTx, tx, NULL);\n",
      "        else {\n",
      "            if(state.IsInvalid())\n",
      "                throw JSONRPCError(RPC_TRANSACTION_REJECTED, strprintf(\"%i: %s\", state.GetRejectCode(), state.GetRejectReason()));\n",
      "            else\n",
      "                throw JSONRPCError(RPC_TRANSACTION_ERROR, state.GetRejectReason());\n",
      "        }\n",
      "    } else if (fHaveChain) {\n",
      "        throw JSONRPCError(RPC_TRANSACTION_ALREADY_IN_CHAIN, \"transaction already in block chain\");\n",
      "    }\n",
      "    RelayTransaction(tx, hashTx);\n",
      "\n",
      "    return hashTx.GetHex();\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "i = 315\n",
    "pii = json.loads(ds[i][\"pii\"])\n",
    "print(\"pii\", pii)\n",
    "print(ds[i][\"content\"][17731:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pii [{'tag': 'PASSWORD', 'value': 'aa7097a2-f2fb-11e7-a565-0a580a28057d', 'start': 1535, 'end': 1571, 'context': \"_at: '2018-01-06T16:11:04.393590+00:00',\\n    id: 'aa7097a2-f2fb-11e7-a565-0a580a28057d',\\n    label: 'fix: Remove break-word behavior on \"}, {'tag': 'SSH_KEY', 'value': 'eff634a68a01d081c0bdc51752dfa0709781f0e4', 'start': 2308, 'end': 2348, 'context': \"break-word behavior on coverage\\\\n',\\n        sha: 'eff634a68a01d081c0bdc51752dfa0709781f0e4'\\n      }\\n    },\\n    started_at: '2018-01-06T16:07\"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 2012, 'end': 2029, 'context': \"  revision: {\\n        author: {\\n          email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28\"}, {'tag': 'PASSWORD', 'value': '63e820d4-81db-11e7-a6df-0a580a28004e', 'start': 3093, 'end': 3129, 'context': \"00',\\n    full_name: 'gh/getsentry/zeus',\\n    id: '63e820d4-81db-11e7-a6df-0a580a28004e',\\n    latest_build: null,\\n    name: 'zeus',\\n    o\"}, {'tag': 'PASSWORD', 'value': 'aa6e1f90-f2fb-11e7-a565-0a580a28057d', 'start': 1919, 'end': 1955, 'context': \"t: '2018-01-06T16:07:16.814650+00:00',\\n      id: 'aa6e1f90-f2fb-11e7-a565-0a580a28057d',\\n      revision: {\\n        author: {\\n          e\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 2103, 'end': 2115, 'context': \"1c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n        committed_at: '2018-01-06T16:\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 2047, 'end': 2083, 'context': \"       email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n     \"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 1744, 'end': 1761, 'context': \"d',\\n    source: {\\n      author: {\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a2804\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 1831, 'end': 1843, 'context': \"c21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      created_at: '2018-01-06T16:07:16.\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 1777, 'end': 1813, 'context': \"\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      cre\"}]\n",
      "eff634a68a01d081c0bdc51752dfa0709781f0e4'\n",
      "      }\n",
      "    },\n",
      "    started_at: '2018-01-06T16:07:16.957093+00:00',\n",
      "    stats: {\n",
      "      coverage: {\n",
      "        diff_lines_covered: 0,\n",
      "        diff_lines_uncovered: 0,\n",
      "        lines_covered: 6127,\n",
      "        lines_uncovered: 3060\n",
      "      },\n",
      "      style_violations: {\n",
      "        count: 0\n",
      "      },\n",
      "      tests: {\n",
      "        count: 153,\n",
      "        count_unique: 153,\n",
      "        duration: 14673.0,\n",
      "        failures: 0,\n",
      "        failures_unique: 0\n",
      "      },\n",
      "      webpack: {\n",
      "        total_asset_size: 0\n",
      "      }\n",
      "    },\n",
      "    status: 'finished',\n",
      "    url: 'https://travis-ci.org/getsentry/zeus/builds/325812408',\n",
      "    ...params\n",
      "  }),\n",
      "\n",
      "  Repository: params => ({\n",
      "    backend: 'git',\n",
      "    created_at: '2017-08-15T17:01:33.206772+00:00',\n",
      "    full_name: 'gh/getsentry/zeus',\n",
      "    id: '63e820d4-81db-11e7-a6df-0a580a28004e',\n",
      "    latest_build: null,\n",
      "    name: 'zeus',\n",
      "    owner_name: 'getsentry',\n",
      "    provider: 'gh',\n",
      "    url: 'git@github.com:getsentry/zeus.git',\n",
      "    permissions: {\n",
      "      admin: true,\n",
      "      read: true,\n",
      "      write: true\n",
      "    },\n",
      "    ...params\n",
      "  })\n",
      "};\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#318\n",
    "i=318\n",
    "pii = json.loads(ds[i][\"pii\"])\n",
    "print(\"pii\", pii)\n",
    "print(ds[i][\"content\"][2308:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pii []\n",
      "qua3H8VlBuvExmhX2b7wKgYO',\n",
      "        'redirect' => 'http://www.gunny.site/callback',\n",
      "    ],\n",
      "];\n"
     ]
    }
   ],
   "source": [
    "i=326\n",
    "pii = json.loads(ds[i][\"pii\"])\n",
    "print(\"pii\", pii)\n",
    "print(ds[i][\"content\"][1141:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pii [{'tag': 'SSH_KEY', 'value': '4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb', 'start': 42, 'end': 106, 'context': 'cask \"tip\" do\\n  version \"2.0.0\"\\n  sha256 \"4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb\"\\n\\n  url \"https://github.com/tanin47/tip/releases/'}]\n",
      "cask \"tip\" do\n",
      "  version \"2.0.0\"\n",
      "  sha256 \"4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb\"\n",
      "\n",
      "  url \"https://github.com/tanin47/tip/releases/download/v#{version}/Tip.zip\"\n",
      "  name \"Tip\"\n",
      "  desc \"Programmable tooltip that can be used with any app\"\n",
      "  homepage \"https://github.com/tanin47/tip\"\n",
      "\n",
      "  app \"Tip.app\"\n",
      "\n",
      "  zap trash: \"~/Library/Application Scripts/tanin.tip\"\n",
      "end\n",
      "\n"
     ]
    }
   ],
   "source": [
    "i=377\n",
    "pii = json.loads(ds[i][\"pii\"])\n",
    "print(\"pii\", pii)\n",
    "print(ds[i][\"content\"][:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add new column with id= row number\n",
    "ds = ds.add_column(\"id\", [i for i in range(len(ds))])\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#include \"googletest/googletest/include/gtest/gtest.h\"\n",
      "\n",
      "#include \"environment.h\"\n",
      "\n",
      "#include \"json/json_spirit_reader_template.h\"\n",
      "#include \"json/json_spirit_utils.h\"\n",
      "#include \"json/json_spirit_writer_template.h\"\n",
      "\n",
      "#include \"base58.h\"\n",
      "#include \"util.h\"\n",
      "\n",
      "using namespace json_spirit;\n",
      "extern Array read_json(const std::string& filename);\n",
      "\n",
      "// Goal: test low-level base58 encoding functionality\n",
      "TEST(base58_tests, base58_EncodeBase58)\n",
      "{\n",
      "    Array tests = read_json(\"base58_encode_decode.json\");\n",
      "\n",
      "    for (Value& tv : tests) {\n",
      "        Array       test    = tv.get_array();\n",
      "        std::string strTest = write_string(tv, false);\n",
      "        if (test.size() < 2) // Allow for extra stuff (useful for comments)\n",
      "        {\n",
      "            ADD_FAILURE() << \"Bad test: \" << strTest;\n",
      "            continue;\n",
      "        }\n",
      "        std::vector<unsigned char> sourcedata   = ParseHex(test[0].get_str());\n",
      "        std::string                base58string = test[1].get_str();\n",
      "        EXPECT_EQ(EncodeBase58(&sourcedata[0], &sourcedata[sourcedata.size()]), base58string) << strTest;\n",
      "    }\n",
      "}\n",
      "\n",
      "// Goal: test low-level base58 decoding functionality\n",
      "TEST(base58_tests, base58_DecodeBase58)\n",
      "{\n",
      "    Array                      tests = read_json(\"base58_encode_decode.json\");\n",
      "    std::vector<unsigned char> result;\n",
      "\n",
      "    BOOST_FOREACH (Value& tv, tests) {\n",
      "        Array       test    = tv.get_array();\n",
      "        std::string strTest = write_string(tv, false);\n",
      "        if (test.size() < 2) // Allow for extra stuff (useful for comments)\n",
      "        {\n",
      "            ADD_FAILURE() << \"Bad test: \" << strTest;\n",
      "            continue;\n",
      "        }\n",
      "        std::vector<unsigned char> expected     = ParseHex(test[0].get_str());\n",
      "        std::string                base58string = test[1].get_str();\n",
      "        EXPECT_TRUE(DecodeBase58(base58string, result)) << strTest;\n",
      "        EXPECT_TRUE(result.size() == expected.size() &&\n",
      "                    std::equal(result.begin(), result.end(), expected.begin()))\n",
      "            << strTest;\n",
      "    }\n",
      "\n",
      "    EXPECT_TRUE(!DecodeBase58(\"invalid\", result));\n",
      "}\n",
      "\n",
      "// Visitor to check address type\n",
      "class TestAddrTypeVisitor : public boost::static_visitor<bool>\n",
      "{\n",
      "private:\n",
      "    std::string exp_addrType;\n",
      "\n",
      "public:\n",
      "    TestAddrTypeVisitor(const std::string& exp_addrTypeIn) : exp_addrType(exp_addrTypeIn) {}\n",
      "    bool operator()(const CKeyID& /*id*/) const { return (exp_addrType == \"pubkey\"); }\n",
      "    bool operator()(const CScriptID& /*id*/) const { return (exp_addrType == \"script\"); }\n",
      "    bool operator()(const CNoDestination& /*no*/) const { return (exp_addrType == \"none\"); }\n",
      "};\n",
      "\n",
      "// Visitor to check address payload\n",
      "class TestPayloadVisitor : public boost::static_visitor<bool>\n",
      "{\n",
      "private:\n",
      "    std::vector<unsigned char> exp_payload;\n",
      "\n",
      "public:\n",
      "    TestPayloadVisitor(std::vector<unsigned char>& exp_payloadIn) : exp_payload(exp_payloadIn) {}\n",
      "    bool operator()(const CKeyID& id) const\n",
      "    {\n",
      "        uint160 exp_key(exp_payload);\n",
      "        return exp_key == id;\n",
      "    }\n",
      "    bool operator()(const CScriptID& id) const\n",
      "    {\n",
      "        uint160 exp_key(exp_payload);\n",
      "        return exp_key == id;\n",
      "    }\n",
      "    bool operator()(const CNoDestination& /*no*/) const { return exp_payload.size() == 0; }\n",
      "};\n",
      "\n",
      "// Goal: check that parsed keys match test payload\n",
      "TEST(base58_tests, base58_keys_valid_parse)\n",
      "{\n",
      "    Array                      tests = read_json(\"base58_keys_valid.json\");\n",
      "    std::vector<unsigned char> result;\n",
      "    CBitcoinSecret             secret;\n",
      "    CBitcoinAddress            addr;\n",
      "\n",
      "    BOOST_FOREACH (Value& tv, tests) {\n",
      "        Array       test    = tv.get_array();\n",
      "        std::string strTest = write_string(tv, false);\n",
      "        if (test.size() < 3) // Allow for extra stuff (useful for comments)\n",
      "        {\n",
      "            ADD_FAILURE() << \"Bad test: \" << strTest;\n",
      "            continue;\n",
      "        }\n",
      "        std::string                  exp_base58string     = test[0].get_str();\n",
      "        std::vector<unsigned char>   privkey_bin_from_hex = ParseHex(test[1].get_str());\n",
      "        const Object&                metadata             = test[2].get_obj();\n",
      "        bool                         isPrivkey            = find_value(metadata, \"isPrivkey\").get_bool();\n",
      "        bool                         isTestnet            = find_value(metadata, \"isTestnet\").get_bool();\n",
      "        SwitchNetworkTypeTemporarily state_holder(isTestnet ? NetworkType::Testnet\n",
      "                                                            : NetworkType::Mainnet);\n",
      "        if (isPrivkey) {\n",
      "            bool isCompressed = find_value(metadata, \"isCompressed\").get_bool();\n",
      "            // Must be valid private key\n",
      "            EXPECT_TRUE(secret.SetString(exp_base58string)) << \"!SetString:\" + strTest;\n",
      "            EXPECT_TRUE(secret.IsValid()) << \"!IsValid:\" + strTest;\n",
      "            bool    fCompressedOut = false;\n",
      "            CSecret privkey        = secret.GetSecret(fCompressedOut);\n",
      "            EXPECT_TRUE(fCompressedOut == isCompressed) << \"compressed mismatch:\" + strTest;\n",
      "            EXPECT_TRUE(privkey.size() == privkey_bin_from_hex.size() &&\n",
      "                        std::equal(privkey.begin(), privkey.end(), privkey_bin_from_hex.begin()))\n",
      "                << \"key mismatch:\" + strTest;\n",
      "\n",
      "            // Private key must be invalid public key\n",
      "            addr.SetString(exp_base58string);\n",
      "            EXPECT_TRUE(!addr.IsValid()) << \"IsValid privkey as pubkey:\" + strTest;\n",
      "        } else {\n",
      "            std::string exp_addrType =\n",
      "                find_value(metadata, \"addrType\").get_str(); // \"script\" or \"pubkey\"\n",
      "            // Must be valid public key\n",
      "            EXPECT_TRUE(addr.SetString(exp_base58string)) << \"SetString:\" + strTest;\n",
      "            EXPECT_TRUE(addr.IsValid()) << \"!IsValid:\" + strTest;\n",
      "            EXPECT_TRUE(addr.IsScript() == (exp_addrType == \"script\")) << \"isScript mismatch\" + strTest;\n",
      "            CTxDestination dest = addr.Get();\n",
      "            EXPECT_TRUE(boost::apply_visitor(TestAddrTypeVisitor(exp_addrType), dest))\n",
      "                << \"addrType mismatch\" + strTest;\n",
      "\n",
      "            // Public key must be invalid private key\n",
      "            secret.SetString(exp_base58string);\n",
      "            EXPECT_TRUE(!secret.IsValid()) << \"IsValid pubkey as privkey:\" + strTest;\n",
      "        }\n",
      "    }\n",
      "}\n",
      "\n",
      "// Goal: check that base58 parsing code is robust against a variety of corrupted data\n",
      "TEST(base58_tests, base58_keys_invalid)\n",
      "{\n",
      "    Array                      tests = read_json(\"base58_keys_invalid.json\"); // Negative testcases\n",
      "    std::vector<unsigned char> result;\n",
      "    CBitcoinSecret             secret;\n",
      "    CBitcoinAddress            addr;\n",
      "\n",
      "    BOOST_FOREACH (Value& tv, tests) {\n",
      "        Array       test    = tv.get_array();\n",
      "        std::string strTest = write_string(tv, false);\n",
      "        if (test.size() < 1) // Allow for extra stuff (useful for comments)\n",
      "        {\n",
      "            ADD_FAILURE() << \"Bad test: \" << strTest;\n",
      "            continue;\n",
      "        }\n",
      "        std::string exp_base58string = test[0].get_str();\n",
      "\n",
      "        // must be invalid as public and as private key\n",
      "        addr.SetString(exp_base58string);\n",
      "        EXPECT_TRUE(!addr.IsValid()) << \"IsValid pubkey:\" + strTest;\n",
      "        secret.SetString(exp_base58string);\n",
      "        EXPECT_TRUE(!secret.IsValid()) << \"IsValid privkey:\" + strTest;\n",
      "    }\n",
      "}\n",
      "\n",
      "template <typename TInputIter>\n",
      "std::string make_hex_string(TInputIter first, TInputIter last, bool use_uppercase = true,\n",
      "                            bool insert_spaces = false)\n",
      "{\n",
      "    std::ostringstream ss;\n",
      "    ss << std::hex << std::setfill('0');\n",
      "    if (use_uppercase)\n",
      "        ss << std::uppercase;\n",
      "    while (first != last) {\n",
      "        ss << std::setw(2) << static_cast<int>(*first++);\n",
      "        if (insert_spaces && first != last)\n",
      "            ss << \" \";\n",
      "    }\n",
      "    return ss.str();\n",
      "}\n",
      "\n",
      "void test_priv_key_vs_address(const std::string& privkey, std::string pubkey, const std::string& address)\n",
      "{\n",
      "    CBitcoinSecret vchSecret;\n",
      "    EXPECT_TRUE(vchSecret.SetString(privkey));\n",
      "    bool    fCompressed;\n",
      "    CKey    key;\n",
      "    CSecret secret = vchSecret.GetSecret(fCompressed);\n",
      "    EXPECT_TRUE(fCompressed);\n",
      "    key.SetSecret(secret, fCompressed);\n",
      "    EXPECT_TRUE(key.GetPubKey().IsValid());\n",
      "    std::vector<unsigned char> rawPubKey = key.GetPubKey().Raw();\n",
      "    std::transform(pubkey.begin(), pubkey.end(), pubkey.begin(), ::tolower); // make pubkey lower-case\n",
      "    EXPECT_EQ(make_hex_string(rawPubKey.begin(), rawPubKey.end(), false), pubkey);\n",
      "    CKeyID keyid = key.GetPubKey().GetID();\n",
      "    EXPECT_EQ(CBitcoinAddress(keyid).ToString(), address);\n",
      "}\n",
      "\n",
      "void test_random_key_generation()\n",
      "{\n",
      "    // create private key from scratch\n",
      "    CKey key;\n",
      "    key.MakeNewKey(true);\n",
      "\n",
      "    // create private key string from raw\n",
      "    CBitcoinSecret vchSecret;\n",
      "    bool           compressed = false;\n",
      "    vchSecret.SetSecret(key.GetSecret(compressed), true);\n",
      "    EXPECT_TRUE(compressed); // currently keys are only compressed\n",
      "\n",
      "    // validate public key\n",
      "    EXPECT_TRUE(key.GetPubKey().IsValid());\n",
      "\n",
      "    // get raw public key\n",
      "    std::vector<unsigned char> rawPubKey = key.GetPubKey().Raw();\n",
      "\n",
      "    // get address\n",
      "    CKeyID          keyid = key.GetPubKey().GetID();\n",
      "    CBitcoinAddress address(keyid);\n",
      "\n",
      "    // test\n",
      "    test_priv_key_vs_address(vchSecret.ToString(),\n",
      "                             make_hex_string(rawPubKey.begin(), rawPubKey.end(), false),\n",
      "                             address.ToString());\n",
      "}\n",
      "\n",
      "TEST(base58_tests, base58_keys_generation)\n",
      "{\n",
      "    {\n",
      "        SwitchNetworkTypeTemporarily state_holder(NetworkType::Mainnet);\n",
      "        // real-net\n",
      "        test_priv_key_vs_address(\"TtnutkcnaPcu3zmjWcrJazf42fp1YAKRpm8grKRRuYjtiykmGuM7\",\n",
      "                                 \"037f41ae8b46979087562e65494eb3a3b9d8addde9b9568ef5cbb8197fd26c0ff2\",\n",
      "                                 \"NVFdK9ik6mBCG6syVw2gD1gBwJzKF5me5i\");\n",
      "        test_priv_key_vs_address(\"TnNwg92Wpw8iuBRwaeJydzw2c6MMqTe2c6cA5hn3NBBdqFWvpViF\",\n",
      "                                 \"03c7f8863df49735b1a1906a5f5939beb8622074b3ddf8fccc5462362271145f09\",\n",
      "                                 \"NSTdV7BgFeYXR61ywiMyAoof2ihwUPDPpj\");\n",
      "        test_priv_key_vs_address(\"TpWmAWxNCGN7tj218djRJjegAVy34K2eEx8Zbt7xbf2H9GNUBgci\",\n",
      "                                 \"029cbf0da830b83a457877fcf009160e9de9f0383fe0c97769ce9a4c2d52949f4b\",\n",
      "                                 \"NdLGazEn51ofFuztenM7bNfquNBV1FWMGG\");\n",
      "\n",
      "        for (int i = 0; i < 10; i++) {\n",
      "            test_random_key_generation();\n",
      "        }\n",
      "    }\n",
      "\n",
      "    {\n",
      "        // test-net\n",
      "        SwitchNetworkTypeTemporarily state_holder(NetworkType::Testnet);\n",
      "\n",
      "        test_priv_key_vs_address(\"Vgg5VL2TW1NMNKt4wkazRkUygpnPiQXnztA2h3ALQxGUhk1tQUag\",\n",
      "                                 \"0243bab8b87abbd42493ca577dee9befc15cc5565f2792e18c52eb90cdfa13dc2a\",\n",
      "                                 \"TVPDsVw4vSbNkRPkfnwCJmbCygEuJEVpwW\");\n",
      "\n",
      "        test_priv_key_vs_address(\"VfCCG4Ew6XAtxpEEnuWTTfUgX92fvyq2kPh7cghHtPDQs7PTmwSn\",\n",
      "                                 \"02f20e5d83d939edc1169296250503b71cc37438ab608fbffb0ea11539d9341c7f\",\n",
      "                                 \"TPiGYtUnB3qCjYBuXAj6QX7CiW5sJQ7Sdk\");\n",
      "\n",
      "        test_priv_key_vs_address(\"VfAy2E8BhcFat6dLGaZotaZJReEU2jWHzZi6a5XQqD9q5qFAWzuK\",\n",
      "                                 \"023ecc7eee129e8b009461b67b9f55120675c61957a7bb7f02726f73215051cb76\",\n",
      "                                 \"THs3Lec52yQPfErz7Z32Yi3KJnBTzicEiz\");\n",
      "\n",
      "        for (int i = 0; i < 10; i++) {\n",
      "            test_random_key_generation();\n",
      "        }\n",
      "    }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(ds[208][\"content\"][:])\n",
    "# delete 208 and replcae it with something else"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "res315 = \"\"\"{'tag': 'KEY',\n",
    "      'value': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.',\n",
    "      'start': 4098,\n",
    "      'end': 4227},\n",
    "     {'tag': 'KEY',\n",
    "      'value': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.',\n",
    "      'start': 4894,\n",
    "      'end': 5023}]}\"\"\"\n",
    "res317 = \"\"\"{'tag': 'KEY',\n",
    "      'value': '12tvKAXCxZjSmdNbao16dKXC8tRWfcF5oc\\\\',\n",
    "      'start': 17731,\n",
    "      'end': 17766},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\',\n",
    "      'start': 9587,\n",
    "      'end': 9624},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\',\n",
    "      'start': 9737,\n",
    "      'end': 9774},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\',\n",
    "      'start': 9544,\n",
    "      'end': 9581},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\',\n",
    "      'start': 9694,\n",
    "      'end': 9731}]}\"\"\"\n",
    "# key res318 should be removed\n",
    "res318 = \"\"\"{'tag': 'KEY',\n",
    "      'value': 'eff634a68a01d081c0bdc51752dfa0709781f0e4',\n",
    "      'start': 2308,\n",
    "      'end': 2348,\n",
    "      'context': \"break-word behavior on coverage\\\\n',\\n        sha: 'eff634a68a01d081c0bdc51752dfa0709781f0e4'\\n      }\\n    },\\n    started_at: '2018-01-06T16:07\"}],\n",
    "    'FP': []}\"\"\"\n",
    "res326 = \"\"\"{'tag': 'KEY',\n",
    "      'value': 'qua3H8VlBuvExmhX2b7wKgYO',\n",
    "      'start': 1141,\n",
    "      'end': 1165}]}\"\"\"\n",
    "res377 = \"\"\"{'tag': 'KEY',\n",
    "      'value': '4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb',\n",
    "      'start': 42,\n",
    "      'end': 106,\n",
    "      'context': 'cask \"tip\" do\\n  version \"2.0.0\"\\n  sha256 \"4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb\"\\n\\n  url \"https://github.com/tanin47/tip/releases/'}],\n",
    "    'FP': []}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_pii(example):\n",
    "    from copy import deepcopy\n",
    "    new_example = deepcopy(example)\n",
    "    pii = json.loads(example[\"pii\"])\n",
    "    if example[\"id\"] == 232:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"43e0352fee07fa5b92dd22e557cb1d050ccde0cf97273e02f694930695b15134\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\")\n",
    "    \n",
    "    elif example[\"id\"] == 239:\n",
    "        # add key\n",
    "        key_1 = {\"tag\": \"KEY\", \"value\": example[\"content\"][5026:5054], \"start\": 5026, \"end\": 5054}\n",
    "        key_2 = {\"tag\": \"KEY\", \"value\": example[\"content\"][10032:10060], \"start\": 10032, \"end\": 10060}\n",
    "        pii.append(key_1)\n",
    "        pii.append(key_2)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\")\n",
    "\n",
    "    elif example[\"id\"] == 269:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"c9eb8a1102d0a68cafc93f22df73445b8f69706f3322285f9a2f623a28df0176\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\")\n",
    "\n",
    "    elif example[\"id\"] == 315:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"KEY\", \"value\": example[\"content\"][17731:17766], \"start\": 17731, \"end\": 17766}\n",
    "        key_2 = {\"tag\": \"KEY\", \"value\": example[\"content\"][9587:9624], \"start\": 9587, \"end\": 9624}\n",
    "        key_3 = {\"tag\": \"KEY\", \"value\": example[\"content\"][9737:9774], \"start\": 9737, \"end\": 9774}\n",
    "        key_4 = {\"tag\": \"KEY\", \"value\": example[\"content\"][9544:9581], \"start\": 9544, \"end\": 9581}\n",
    "        key_5 = {\"tag\": \"KEY\", \"value\": example[\"content\"][9694:9731], \"start\": 9694, \"end\": 9731}\n",
    "        pii.append(key_1)\n",
    "        pii.append(key_2)\n",
    "        pii.append(key_3)\n",
    "        pii.append(key_4)\n",
    "        pii.append(key_5)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\")\n",
    "\n",
    "    elif example[\"id\"] == 317:\n",
    "        # add 5 missing keys\n",
    "        key_1 = {\"tag\": \"KEY\", \"value\": example[\"content\"][4098:4227], \"start\": 4098, \"end\": 4227}\n",
    "        key_2 = {\"tag\": \"KEY\", \"value\": example[\"content\"][4894:5023], \"start\": 4894, \"end\": 5023}\n",
    "        pii.append(key_1)\n",
    "        pii.append(key_2)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "\n",
    "    elif example[\"id\"] == 318:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"eff634a68a01d081c0bdc51752dfa0709781f0e4\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\")\n",
    "    \n",
    "    elif example[\"id\"] == 326:\n",
    "        # add missing key\n",
    "        key_1 = {\"tag\": \"KEY\", \"value\": example[\"content\"][1141:1165], \"start\": 1141, \"end\": 1165}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\")\n",
    "\n",
    "    elif example[\"id\"] == 377:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"4d986a461d1b24bb5776fb49063b9a1891939f336b306a6bc75f58d0a4e98bcb\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\")\n",
    "    \n",
    "    return new_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ad2d63b9e024da8b79efd97bb9ac7f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/400 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new pii for 232: [{'tag': 'USERNAME', 'value': 'WebStorm', 'start': 214, 'end': 222, 'context': 'bStorm-#{version}-custom-jdk-bundled.dmg\"\\n  name \\'WebStorm\\'\\n  homepage \\'http://www.jetbrains.com/webstorm/\\'\\n'}]\n",
      "new pii for 239: [{'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 7889, 'end': 7897, 'context': 'n/json\")\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        b = client._get(\"/bar\", {}, extract_bod'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 6861, 'end': 6869, 'context': 'n/json\\')\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        with self.assertRaises(googlemaps.excep'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 2938, 'end': 2946, 'context': 'i/geocode/json?\"\\n                            \"key=AIzaasdf&address=Sesame+St.\",\\n                            '}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 2107, 'end': 2115, 'context': 'on/json\")\\n        client = googlemaps.Client(key=\"AIzaasdf\",\\n                                   queries_per_'}, {'tag': 'API_KEY', 'value': '0xde7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9', 'start': 3932, 'end': 3974, 'context': ' brown fox jumps over the lazy dog\")\\n           = 0xde7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9\\n        \"\"\"\\n\\n        message = \"The quick brown f'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 10382, 'end': 10390, 'context': 'self.assertEqual(auth_url, \"/test?param=param&key=AIzaasdf\")\\n\\n    def test_requests_version(self):\\n        c'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 7362, 'end': 7370, 'context': 'n/json\")\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        client._get(\"/bar\", {}, base_url=\"https'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 2725, 'end': 2733, 'context': 'n/json\")\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        client.geocode(\"Sesame St.\")\\n\\n        s'}, {'tag': 'AMBIGUOUS', 'value': 'a2V5', 'start': 9274, 'end': 9278, 'context': 'googlemaps.Client(client_id=\"foo\", client_secret=\"a2V5\",\\n                                       channel='}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 10483, 'end': 10491, 'context': '      client_args_timeout = {\\n            \"key\": \"AIzaasdf\",\\n            \"client_id\": \"foo\",\\n            \"cl'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 11479, 'end': 11487, 'context': 'n/json\")\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\",\\n                                   retry_over_q'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 3387, 'end': 3395, 'context': 'n/json\")\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        client.geocode(\"Sesame St.\", extra_para'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 9426, 'end': 9434, 'context': 'el(self):\\n        client = googlemaps.Client(key=\"AIzaasdf\",\\n                                   client_id=\"f'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 8963, 'end': 8971, 'context': 'ror):\\n            client = googlemaps.Client(key=\"AIzaasdf\", channel=\"mychannel\")\\n\\n    def test_invalid_chan'}, {'tag': 'AMBIGUOUS', 'value': 'fxbWUIcNPZSekVOhp2ul9LW5TpY=', 'start': 5026, 'end': 5054, 'context': 'ient=foo&\"\\n                            \"signature=fxbWUIcNPZSekVOhp2ul9LW5TpY=\",\\n                            responses.calls[0].'}, {'tag': 'AMBIGUOUS', 'value': 'OH18GuQto_mEpxj99UimKskvo4k=', 'start': 10032, 'end': 10060, 'context': 'ient=foo\"\\n                            \"&signature=OH18GuQto_mEpxj99UimKskvo4k=\")\\n\\n        # Check if added to requests to API wi'}, {'tag': 'AMBIGUOUS', 'value': '3nybhbi3iqa8ino29wqQcBydtNk=', 'start': 4112, 'end': 4140, 'context': 'y = \"a2V5\" # \"key\" -> base64\\n        signature = \"3nybhbi3iqa8ino29wqQcBydtNk=\"\\n\\n        self.assertEqual(signature, _client.sig'}, {'tag': 'PASSWORD', 'value': 'AIzaasdf', 'start': 8729, 'end': 8737, 'context': 'lback())\\n\\n        client = googlemaps.Client(key=\"AIzaasdf\")\\n        client.geocode(\"Sesame St.\")\\n\\n        s'}, {'tag': 'KEY', 'value': 'fxbWUIcNPZSekVOhp2ul9LW5TpY=', 'start': 5026, 'end': 5054}, {'tag': 'KEY', 'value': 'OH18GuQto_mEpxj99UimKskvo4k=', 'start': 10032, 'end': 10060}]\n",
      "new pii for 269: [{'tag': 'USERNAME', 'value': 'cdrummond', 'start': 322, 'end': 331, 'context': \"'\\n  name 'Cantata'\\n  homepage 'https://github.com/cdrummond/cantata'\\n\\n  depends_on macos: '>= :sierra'\\n\\n  app\"}, {'tag': 'USERNAME', 'value': 'CDrummond', 'start': 139, 'end': 148, 'context': '2285f9a2f623a28df0176\\'\\n\\n  url \"https://github.com/CDrummond/cantata/releases/download/v#{version}/Cantata-#{v'}, {'tag': 'USERNAME', 'value': 'CDrummond', 'start': 241, 'end': 250, 'context': 'ata-#{version}.dmg\"\\n  appcast \\'https://github.com/CDrummond/cantata/releases.atom\\'\\n  name \\'Cantata\\'\\n  homepag'}]\n",
      "new pii for 315: [{'tag': 'NAME', 'value': 'Satoshi Nakamoto', 'start': 22, 'end': 38, 'context': '// Copyright (c) 2010 Satoshi Nakamoto\\n// Original Code: Copyright (c) 2009-2014 The Bit'}, {'tag': 'KEY', 'value': '12tvKAXCxZjSmdNbao16dKXC8tRWfcF5oc\\\\', 'start': 17731, 'end': 17766}, {'tag': 'KEY', 'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\', 'start': 9587, 'end': 9624}, {'tag': 'KEY', 'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\', 'start': 9737, 'end': 9774}, {'tag': 'KEY', 'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\', 'start': 9544, 'end': 9581}, {'tag': 'KEY', 'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\', 'start': 9694, 'end': 9731}]\n",
      "new pii for 318: [{'tag': 'PASSWORD', 'value': 'aa7097a2-f2fb-11e7-a565-0a580a28057d', 'start': 1535, 'end': 1571, 'context': \"_at: '2018-01-06T16:11:04.393590+00:00',\\n    id: 'aa7097a2-f2fb-11e7-a565-0a580a28057d',\\n    label: 'fix: Remove break-word behavior on \"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 2012, 'end': 2029, 'context': \"  revision: {\\n        author: {\\n          email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28\"}, {'tag': 'PASSWORD', 'value': '63e820d4-81db-11e7-a6df-0a580a28004e', 'start': 3093, 'end': 3129, 'context': \"00',\\n    full_name: 'gh/getsentry/zeus',\\n    id: '63e820d4-81db-11e7-a6df-0a580a28004e',\\n    latest_build: null,\\n    name: 'zeus',\\n    o\"}, {'tag': 'PASSWORD', 'value': 'aa6e1f90-f2fb-11e7-a565-0a580a28057d', 'start': 1919, 'end': 1955, 'context': \"t: '2018-01-06T16:07:16.814650+00:00',\\n      id: 'aa6e1f90-f2fb-11e7-a565-0a580a28057d',\\n      revision: {\\n        author: {\\n          e\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 2103, 'end': 2115, 'context': \"1c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n        committed_at: '2018-01-06T16:\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 2047, 'end': 2083, 'context': \"       email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n     \"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 1744, 'end': 1761, 'context': \"d',\\n    source: {\\n      author: {\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a2804\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 1831, 'end': 1843, 'context': \"c21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      created_at: '2018-01-06T16:07:16.\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 1777, 'end': 1813, 'context': \"\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      cre\"}]\n",
      "new pii for 326: [{'tag': 'KEY', 'value': 'qua3H8VlBuvExmhX2b7wKgYO', 'start': 1141, 'end': 1165}]\n",
      "new pii for 377: []\n"
     ]
    }
   ],
   "source": [
    "new_ds = ds.map(update_pii)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'indices'=<generator object <genexpr> at 0x7f94603c69d0> of the transform datasets.arrow_dataset.Dataset.select couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n"
     ]
    }
   ],
   "source": [
    "one_sample = dataset_no_filter.select([17])\n",
    "hf_ds = new_ds.select((i for i in range(len(new_ds)) if i!=208))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 399\n",
       "})"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hf_ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_clean = datasets.concatenate_datasets([hf_ds, one_sample])\n",
    "ds_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "659140a787ac4ca9911deb48fa939b5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Pushing dataset shards to the dataset hub:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "44e96507cc2c4fc196c184de630e60da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Deleting unused files from dataset repository:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e13b783da9984e4a884ce55294153ef3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading metadata:   0%|          | 0.00/454 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Updating downloaded metadata with the new split.\n"
     ]
    }
   ],
   "source": [
    "ds_clean.push_to_hub(\"dummy_data2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65d2fe77f0a54d37aead5fd1c2230112",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3724499"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_clean.to_json(\"prefiltered_data.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration data-490d13ba0a302c92\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset json/data to /Users/loubnabenallal/.cache/huggingface/datasets/json/data-490d13ba0a302c92/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97b5168210294e598db94ae6faa244da",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ceb08e85c7bc493da77241b0273ebf37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3b8467257984007b6f2d26601f81e41",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0 tables [00:00, ? tables/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset json downloaded and prepared to /Users/loubnabenallal/.cache/huggingface/datasets/json/data-490d13ba0a302c92/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31b097a0c80247f18a07296bf2a5c4ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dd = load_dataset(\"pii-for-code/data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'content': '\"use strict\";\\n\\nvar _interopRequireDefault = require(\"@babel/runtime/helpers/builtin/interopRequireDefault\");\\n\\nObject.defineProperty(exports, \"__esModule\", {\\n  value: true\\n});\\nexports.default = void 0;\\n\\nvar _react = _interopRequireDefault(require(\"react\"));\\n\\nvar _createSvgIcon = _interopRequireDefault(require(\"./utils/createSvgIcon\"));\\n\\nvar _default = (0, _createSvgIcon.default)(_react.default.createElement(_react.default.Fragment, null, _react.default.createElement(\"path\", {\\n  d: \"M9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4zm2 2H5V5h14v14zm2-16H3v18h18V3z\"\\n})), \\'InsertChartOutlinedSharp\\');\\n\\nexports.default = _default;',\n",
       " 'language': 'JavaScript',\n",
       " 'license': 'MIT',\n",
       " 'path': 'GJCHOWDARY/chowdary_react/node_modules/@material-ui/icons/InsertChartOutlinedSharp.js',\n",
       " 'annotation_id': '5157a4f9-7197-402f-807d-f944390b0334',\n",
       " 'pii': '[]',\n",
       " 'pii_modified': '[]',\n",
       " 'id': 209.0}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dd[\"train\"][208]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration bigcode--pii-for-code-a9a3a14b9b71dff0\n",
      "Found cached dataset json (/Users/loubnabenallal/.cache/huggingface/datasets/bigcode___json/bigcode--pii-for-code-a9a3a14b9b71dff0/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"bigcode/pii-for-code\", use_auth_token=True, split=\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add id column to dataset as a new column\n",
    "ds = ds.remove_columns([\"id\"])\n",
    "ds = ds.add_column(\"id\", [i for i in range(len(ds))])\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# remove key\n",
    "res73 = \"\"\"'FN': [{'tag': 'KEY',\n",
    "      'value': '6622386f8d83dc9efefb8c03a4dbfc18e7928d89ffc2ec3e2feb9473e8f410c9',\n",
    "      'start': 57,\n",
    "      'end': 121,\n",
    "      'context': \"ockrattler' do\\n  version '4.15,2018.11'\\n  sha256 '6622386f8d83dc9efefb8c03a4dbfc18e7928d89ffc2ec3e2feb9473e8f410c9'\\n\\n  # eclecticlightdotcom.files.wordpress.com was\"}],\n",
    "    'FP': []}}}\"\"\"\n",
    "\n",
    "\n",
    "res365 = \"\"\"'FP': [{'tag': 'KEY',\n",
    "      'value': '23fedee89bcadec0487bf990c2c714d1',\n",
    "      'start': 3116,\n",
    "      'end': 3148}]}}}\"\"\"\n",
    "\n",
    "# remove key\n",
    "res137 = \"\"\"'FN': [{'tag': 'KEY',\n",
    "      'value': '57ed642b-1ee3-47b3-be6d-b7297d500409',\n",
    "      'start': 3842,\n",
    "      'end': 3878,\n",
    "      'context': 'nal UUID SCHEDULER_STORE_TOKEN = UUID.fromString(\"57ed642b-1ee3-47b3-be6d-b7297d500409\");\\n\\n    /**\\n     * The default scheduler store ve'}],\n",
    "    'FP': []}}}\"\"\"\n",
    "\n",
    "#remove key\n",
    "res188 = \"\"\"'FN': [{'tag': 'KEY',\n",
    "      'value': '546d57b6c88c2be7517759c016c0bf0313dfcc14adfcb43967f3c5d24657f366',\n",
    "      'start': 61,\n",
    "      'end': 125,\n",
    "      'context': 'owser-for-sqlite\" do\\n  version \"3.12.2\"\\n  sha256 \"546d57b6c88c2be7517759c016c0bf0313dfcc14adfcb43967f3c5d24657f366\"\\n\\n  url \"https://github.com/sqlitebrowser/sqliteb'}]\"\"\"\n",
    "\n",
    "#remove key\n",
    "res221 = \"\"\"'FN': [{'tag': 'KEY',\n",
    "      'value': '76d8ae334545bbdf2db49414c25d2cfd8685e7b6187f119b28e93ad9c5118e9d',\n",
    "      'start': 54,\n",
    "      'end': 118,\n",
    "      'context': 'ion https://git-lfs.github.com/spec/v1\\noid sha256:76d8ae334545bbdf2db49414c25d2cfd8685e7b6187f119b28e93ad9c5118e9d\\nsize 4727\\n'}]\"\"\"\n",
    "\n",
    "res316 = \"\"\"    'FP': [{'tag': 'EMAIL',\n",
    "      'value': 'fake@example.com',\n",
    "      'start': 1796,\n",
    "      'end': 1812},\n",
    "     {'tag': 'EMAIL', 'value': 'fake@example.com', 'start': 2904, 'end': 2920},\n",
    "     {'tag': 'EMAIL',\n",
    "      'value': 'fake@example.com',\n",
    "      'start': 5893,\n",
    "      'end': 5909}]}\"\"\"\n",
    "\n",
    "res317 = \"\"\"    'FP': [{'tag': 'EMAIL',\n",
    "      'value': 'git@github.com',\n",
    "      'start': 3233,\n",
    "      'end': 3247}]}\"\"\"\n",
    "\n",
    "res346 = \"\"\"    'FP': [{'tag': 'EMAIL',\n",
    "      'value': 'oli@dwslab.de',\n",
    "      'start': 1971,\n",
    "      'end': 1984}]}\"\"\"\n",
    "\n",
    "res355 = \"\"\"'FP': [{'tag': 'EMAIL',\n",
    "      'value': 'license@prestashop.com',\n",
    "      'start': 435,\n",
    "      'end': 457}]}\"\"\"\n",
    "\n",
    "res362 = \"\"\"    'FP': [{'tag': 'EMAIL',\n",
    "      'value': 'bootrax@rackspace.com',\n",
    "      'start': 3785,\n",
    "      'end': 3806}]}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token || '23fedee89bcadec0487bf990c2c714d1',\n",
      "        });\n",
      "\n",
      "\n",
      "        // Sample Options:\n",
      "        var defaultOptions = {\n",
      "            \"product\": {\n",
      "                \"layout\": \"horizontal\",\n",
      "                    \"variantId\": \"all\",\n",
      "                    \"width\": \"100%\",\n",
      "                    \"contents\": {\n",
      "                    \"img\": false,\n",
      "                        \"imgWithCarousel\": true,\n",
      "                        \"variantTitle\": false,\n",
      "                        \"description\": true,\n",
      "                        \"buttonWithQuantity\": false,\n",
      "                        \"quantity\": false\n",
      "                },\n",
      "                \"styles\": {\n",
      "                    \"product\": {\n",
      "                        \"text-align\": \"left\",\n",
      "                            \"@media (min-width: 601px)\": {\n",
      "                            \"max-width\": \"100%\",\n",
      "                                \"margin-left\": \"0\",\n",
      "                                \"margin-bottom\": \"50px\"\n",
      "                        }\n",
      "                    },\n",
      "                    \"title\": {\n",
      "                        \"font-size\": \"26px\"\n",
      "                    },\n",
      "                    \"price\": {\n",
      "                        \"font-size\": \"18px\"\n",
      "                    },\n",
      "                    \"compareAt\": {\n",
      "                        \"font-size\": \"15px\"\n",
      "                    }\n",
      "                }\n",
      "            },\n",
      "            \"cart\": {\n",
      "                \"contents\": {\n",
      "                    \"button\": true\n",
      "                },\n",
      "                \"styles\": {\n",
      "                    \"footer\": {\n",
      "                        \"background-color\": \"#ffffff\"\n",
      "                    }\n",
      "                }\n",
      "            },\n",
      "            \"modalProduct\": {\n",
      "                \"contents\": {\n",
      "                    \"img\": false,\n",
      "                        \"imgWithCarousel\": true,\n",
      "                        \"variantTitle\": false,\n",
      "                        \"buttonWithQuantity\": true,\n",
      "                        \"button\": false,\n",
      "                        \"quantity\": false\n",
      "                },\n",
      "                \"styles\": {\n",
      "                    \"product\": {\n",
      "                        \"@media (min-width: 601px)\": {\n",
      "                            \"max-width\": \"100%\",\n",
      "                                \"margin-left\": \"0px\",\n",
      "                                \"margin-bottom\": \"0px\"\n",
      "                        }\n",
      "                    }\n",
      "                }\n",
      "            },\n",
      "            \"productSet\": {\n",
      "                \"styles\": {\n",
      "                    \"products\": {\n",
      "                        \"@media (min-width: 601px)\": {\n",
      "                            \"margin-left\": \"-20px\"\n",
      "                        }\n",
      "                    }\n",
      "                }\n",
      "            }\n",
      "        };\n",
      "        ShopifyBuy.UI.onReady(this.client).then(function (ui) {\n",
      "            ui.createComponent('product', {\n",
      "                id: idsArray || [1658849624164],\n",
      "                node: document.getElementById('shopify-container'),\n",
      "                moneyFormat: '%24%7B%7Bamount%7D%7D',\n",
      "                options: options || defaultOptions\n",
      "            });\n",
      "        });\n",
      "    },\n",
      "    addProuct: function addProduct() {\n",
      "\n",
      "    },\n",
      "    trackAction: function hyTrackAction (action, mode, values) {\n",
      "        var dotget = this.library.utils.dotget;\n",
      "\n",
      "        var data;\n",
      "\n",
      "        if(!action || !mode || !values) {\n",
      "            return;\n",
      "        }\n",
      "\n",
      "        data = $.extend({}, values);\n",
      "\n",
      "        this.library.utils.track.event('hapyak', action, data);\n",
      "    },\n",
      "    loadScript: function loadScript() {\n",
      "        var script = document.createElement('script'),\n",
      "            scriptURL = 'https://sdks.shopifycdn.com/buy-button/latest/buy-button-storefront.min.js';\n",
      "\n",
      "        script.async = true;\n",
      "        script.src = scriptURL;\n",
      "        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(script);\n",
      "        script.onload = function () {\n",
      "            window.ShopifyBuy && window.ShopifyBuy.UI ? hapyak.shopifyWidget.shopifyLoaded = true : hapyak.shopifyWidget.shopifyLoaded = false;\n",
      "        };\n",
      "    },\n",
      "    customLoad: function customLoad() {\n",
      "        /*\n",
      "            Required to init widget load for both editor and viewer.\n",
      "            Widgets may require unique events to occur before load, so this logic\n",
      "            is executed on a per widget basis.\n",
      "        */\n",
      "\n",
      "        window.ShopifyBuy && window.ShopifyBuy.UI ? hapyak.shopifyWidget.shopifyLoaded = true : hapyak.shopifyWidget.loadScript();\n",
      "\n",
      "        hapyak.widget.library.utils.startLoad();\n",
      "\n",
      "    }\n",
      "};\n",
      "\n",
      "hapyak.widget.library.utils.onWidgetLoad(hapyak.shopifyWidget.init.bind(hapyak.shopifyWidget));\n",
      "hapyak.context.addEventListener('iframeshow', hapyak.shopifyWidget.customLoad, false);\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(ds[365][\"content\"][3106:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_pii(example):\n",
    "    from copy import deepcopy\n",
    "    new_example = deepcopy(example)\n",
    "    pii = json.loads(example[\"pii\"])\n",
    "    if example[\"id\"] == 73:\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"6622386f8d83dc9efefb8c03a4dbfc18e7928d89ffc2ec3e2feb9473e8f410c9\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 137:\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"57ed642b-1ee3-47b3-be6d-b7297d500409\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 188:\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"546d57b6c88c2be7517759c016c0bf0313dfcc14adfcb43967f3c5d24657f366\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 221:\n",
    "        new_pii = [key for key in pii if key[\"value\"] != \"76d8ae334545bbdf2db49414c25d2cfd8685e7b6187f119b28e93ad9c5118e9d\"]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 355:\n",
    "        key_1 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][435:457], \"start\": 435, \"end\": 457}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 365:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"KEY\", \"value\": example[\"content\"][3116:3148], \"start\": 3116, \"end\": 3148}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 316:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][1796:1812], \"start\": 1796, \"end\": 1812}\n",
    "        key_2 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][2904:2920], \"start\": 2904, \"end\": 2920}\n",
    "        key_3 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][5893:5909], \"start\": 5893, \"end\": 5909}\n",
    "        pii.append(key_1)\n",
    "        pii.append(key_2)\n",
    "        pii.append(key_3)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 317:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][3233:3247], \"start\": 3233, \"end\": 3247}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 346:\n",
    "        key_1 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][1971:1984], \"start\": 1971, \"end\": 1984}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 362:\n",
    "        key_1 = {\"tag\": \"EMAIL\", \"value\": example[\"content\"][3785:3806], \"start\": 3785, \"end\": 3806}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    return new_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc3f83d0a9914dc6846aa8a107f420a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/400 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new pii for 73: []\n",
      "\n",
      "new pii for 137: []\n",
      "\n",
      "new pii for 188: []\n",
      "\n",
      "new pii for 221: []\n",
      "\n",
      "new pii for 316: [{'tag': 'KEY', 'value': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.', 'start': 4098, 'end': 4227}, {'tag': 'KEY', 'value': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI1ZWJhYzUzNDk1NGI1NDEzOTgwNmMxMTIiLCJpYXQiOjE1ODkyOTg0ODQsImV4cCI6MTU4OTMwMDI4NH0.', 'start': 4894, 'end': 5023}, {'tag': 'EMAIL', 'value': 'fake@example.com', 'start': 1796, 'end': 1812}, {'tag': 'EMAIL', 'value': 'fake@example.com', 'start': 2904, 'end': 2920}, {'tag': 'EMAIL', 'value': 'fake@example.com', 'start': 5893, 'end': 5909}]\n",
      "\n",
      "new pii for 317: [{'tag': 'PASSWORD', 'value': 'aa7097a2-f2fb-11e7-a565-0a580a28057d', 'start': 1535, 'end': 1571, 'context': \"_at: '2018-01-06T16:11:04.393590+00:00',\\n    id: 'aa7097a2-f2fb-11e7-a565-0a580a28057d',\\n    label: 'fix: Remove break-word behavior on \"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 2012, 'end': 2029, 'context': \"  revision: {\\n        author: {\\n          email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28\"}, {'tag': 'PASSWORD', 'value': '63e820d4-81db-11e7-a6df-0a580a28004e', 'start': 3093, 'end': 3129, 'context': \"00',\\n    full_name: 'gh/getsentry/zeus',\\n    id: '63e820d4-81db-11e7-a6df-0a580a28004e',\\n    latest_build: null,\\n    name: 'zeus',\\n    o\"}, {'tag': 'PASSWORD', 'value': 'aa6e1f90-f2fb-11e7-a565-0a580a28057d', 'start': 1919, 'end': 1955, 'context': \"t: '2018-01-06T16:07:16.814650+00:00',\\n      id: 'aa6e1f90-f2fb-11e7-a565-0a580a28057d',\\n      revision: {\\n        author: {\\n          e\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 2103, 'end': 2115, 'context': \"1c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n        committed_at: '2018-01-06T16:\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 2047, 'end': 2083, 'context': \"       email: 'dcramer@gmail.com',\\n          id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n          name: 'David Cramer'\\n        },\\n     \"}, {'tag': 'EMAIL', 'value': 'dcramer@gmail.com', 'start': 1744, 'end': 1761, 'context': \"d',\\n    source: {\\n      author: {\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a2804\"}, {'tag': 'NAME', 'value': 'David Cramer', 'start': 1831, 'end': 1843, 'context': \"c21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      created_at: '2018-01-06T16:07:16.\"}, {'tag': 'PASSWORD', 'value': '659dc21c-81db-11e7-988a-0a580a28047a', 'start': 1777, 'end': 1813, 'context': \"\\n        email: 'dcramer@gmail.com',\\n        id: '659dc21c-81db-11e7-988a-0a580a28047a',\\n        name: 'David Cramer'\\n      },\\n      cre\"}, {'tag': 'EMAIL', 'value': 'git@github.com', 'start': 3233, 'end': 3247}]\n",
      "\n",
      "new pii for 346: [{'tag': 'EMAIL', 'value': 'oli@dwslab.de', 'start': 1971, 'end': 1984}]\n",
      "\n",
      "new pii for 355: [{'tag': 'EMAIL', 'value': 'contact@prestashop.com', 'start': 772, 'end': 794, 'context': 'more information.\\n *\\n * @author    PrestaShop SA <contact@prestashop.com>\\n * @copyright 2007-2018 PrestaShop SA\\n * @licens'}, {'tag': 'EMAIL', 'value': 'license@prestashop.com', 'start': 435, 'end': 457}]\n",
      "\n",
      "new pii for 362: [{'tag': 'USERNAME', 'value': '@ajmesserli', 'start': 3960, 'end': 3971, 'context': ' Engineering team. You can follow him on twitter [@ajmesserli](http://twitter.com/ajmesserli) and on Github as '}, {'tag': 'USERNAME', 'value': 'amesserl', 'start': 4051, 'end': 4059, 'context': 'i) and on Github as [amesserl](https://github.com/amesserl).\\n'}, {'tag': 'USERNAME', 'value': 'amesserl', 'start': 4022, 'end': 4030, 'context': '(http://twitter.com/ajmesserli) and on Github as [amesserl](https://github.com/amesserl).\\n'}, {'tag': 'USERNAME', 'value': 'ajmesserli', 'start': 3992, 'end': 4002, 'context': 'w him on twitter [@ajmesserli](http://twitter.com/ajmesserli) and on Github as [amesserl](https://github.com/a'}, {'tag': 'NAME', 'value': 'Antony Messerli', 'start': 3829, 'end': 3844, 'context': 's at <bootrax@rackspace.com>.\\n\\n# About the Author\\nAntony Messerli is a Principal Engineer at Rackspace working on t'}, {'tag': 'NAME', 'value': 'Antony Messerli', 'start': 103, 'end': 118, 'context': 'om\"\\ndate: 2014-01-23 16:00\\ncomments: true\\nauthor: Antony Messerli\\npublished: true\\ncategories: \\n- Cloud Servers\\n- Cl'}, {'tag': 'EMAIL', 'value': 'bootrax@rackspace.com', 'start': 3785, 'end': 3806}]\n",
      "\n",
      "new pii for 365: [{'tag': 'KEY', 'value': '23fedee89bcadec0487bf990c2c714d1', 'start': 3116, 'end': 3148}]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "new_ds = ds.map(update_pii, load_from_cache_file=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "res16 = \"\"\"    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '72.44.36.12',\n",
    "      'start': 1193,\n",
    "      'end': 1204}]}\"\"\"\n",
    "\n",
    "res60 = \"\"\"   'IP_ADDRESS': {'FN': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '224.250.0.2',\n",
    "      'start': 1600,\n",
    "      'end': 1611,\n",
    "      'context': ';239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%en0]:7500\\nint'},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '224.250.0.3:8000',\n",
    "      'start': 1612,\n",
    "      'end': 1628,\n",
    "      'context': ':7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%en0]:7500\\nint zmq::pgm_socket_'},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '239.192.0.1:7500',\n",
    "      'start': 1551,\n",
    "      'end': 1567,\n",
    "      'context': ' & multicast group decls>:<IP port>\\n//  e.g. eth0;239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.2'}],\n",
    "    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '239.192.0.1',\n",
    "      'start': 1551,\n",
    "      'end': 1562},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '224.250.0.3',\n",
    "      'start': 1612,\n",
    "      'end': 1623}]}\"\"\"\n",
    "\n",
    "res70 = \"\"\"    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '87.248.113.14',\n",
    "      'start': 6973,\n",
    "      'end': 6986}]}\"\"\"\n",
    "\n",
    "res189 = \"\"\"    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': 'FF02::FB',\n",
    "      'start': 3326,\n",
    "      'end': 3334},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '224.0.0.251',\n",
    "      'start': 3536,\n",
    "      'end': 3547}]}\"\"\"\n",
    "\n",
    "res214 = \"\"\"    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '127.0.0.1',\n",
    "      'start': 293,\n",
    "      'end': 302}]}\"\"\"\n",
    "\n",
    "res254 = \"\"\"    'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '192.168.0.9',\n",
    "      'start': 1514,\n",
    "      'end': 1525}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_pii_2(example):\n",
    "    from copy import deepcopy\n",
    "    new_example = deepcopy(example)\n",
    "    pii = json.loads(example[\"pii\"])\n",
    "    if example[\"id\"] == 16:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][1193:1204], \"start\": 1193, \"end\": 1204}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 70:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][6973:6986], \"start\": 6973, \"end\": 6986}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 189:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][3326:3334], \"start\": 3326, \"end\": 3334}\n",
    "        key_2 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][3536:3547], \"start\": 3536, \"end\": 3547}\n",
    "        pii.append(key_1)\n",
    "        pii.append(key_2)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 214:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][293:302], \"start\": 293, \"end\": 302}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 254:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][1514:1525], \"start\": 1514, \"end\": 1525}\n",
    "        pii.append(key_1)\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 60:\n",
    "        # remove key\n",
    "        # get key that has the value \"224.250.0.3:8000\"\n",
    "        key_1 = [key for key in pii if key[\"value\"] == \"224.250.0.3:8000\"][0]\n",
    "        key_2 = [key for key in pii if key[\"value\"] == \"239.192.0.1:7500\"][0]\n",
    "        key_1[\"value\"] =  \"224.250.0.3\"\n",
    "        key_2[\"value\"] = \"239.192.0.1\"\n",
    "        key_1[\"end\"] = 1623\n",
    "        key_2[\"end\"] = 1562\n",
    "        new_pii = [x for x in pii if x[\"value\"] not in [\"224.250.0.3:8000\", \"239.192.0.1:7500\"]]\n",
    "        new_pii.extend((key_1, key_2))\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    return new_example        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "786678d330a6459c8eb97fa9c2ba9245",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/400 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new pii for 16: [{'tag': 'IP_ADDRESS', 'value': '72.44.36.12', 'start': 1336, 'end': 1347, 'context': '  \"72.44.36.12\".convert_from_ec2_to_ip.should == \"72.44.36.12\"\\n    end\\n    it \"should be able to parse the date'}, {'tag': 'IP_ADDRESS', 'value': '72.44.36.12', 'start': 1289, 'end': 1300, 'context': 'an error if another string is returned\" do\\n      \"72.44.36.12\".convert_from_ec2_to_ip.should == \"72.44.36.12\"\\n '}, {'tag': 'IP_ADDRESS', 'value': '72.44.36.12', 'start': 1193, 'end': 1204}]\n",
      "\n",
      "new pii for 60: [{'tag': 'IP_ADDRESS', 'value': '224.250.0.1', 'start': 1588, 'end': 1599, 'context': '/  e.g. eth0;239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%e'}, {'tag': 'IP_ADDRESS', 'value': '224.250.0.2', 'start': 1600, 'end': 1611, 'context': ';239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%en0]:7500\\nint'}, {'tag': 'IP_ADDRESS', 'value': '224.250.0.3', 'start': 1612, 'end': 1623, 'context': ':7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%en0]:7500\\nint zmq::pgm_socket_'}, {'tag': 'IP_ADDRESS', 'value': '239.192.0.1', 'start': 1551, 'end': 1562, 'context': ' & multicast group decls>:<IP port>\\n//  e.g. eth0;239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.2'}, {'tag': 'IP_ADDRESS', 'value': '224.250.0.3', 'start': 1612, 'end': 1623, 'context': ':7500\\n//       link-local;224.250.0.1,224.250.0.2;224.250.0.3:8000\\n//       ;[fe80::1%en0]:7500\\nint zmq::pgm_socket_'}, {'tag': 'IP_ADDRESS', 'value': '239.192.0.1', 'start': 1551, 'end': 1562, 'context': ' & multicast group decls>:<IP port>\\n//  e.g. eth0;239.192.0.1:7500\\n//       link-local;224.250.0.1,224.250.0.2;224.2'}]\n",
      "\n",
      "new pii for 70: [{'tag': 'IP_ADDRESS', 'value': '87.248.113.14', 'start': 2415, 'end': 2428, 'context': 'ttpCallBean2 = cougarManager2.getNewHttpCallBean(\"87.248.113.14\");\\n        cougarManager2 = cougarManager2;\\n\\n    '}, {'tag': 'IP_ADDRESS', 'value': '87.248.113.14', 'start': 6881, 'end': 6894, 'context': 'eration\", \"BadRequest\"),new AccessLogRequirement(\"87.248.113.14\", \"/cougarBaseline/v2/enumOperation\", \"BadRequest'}, {'tag': 'IP_ADDRESS', 'value': '87.248.113.14', 'start': 6789, 'end': 6802, 'context': 'e(getTimeAsTimeStamp11, new AccessLogRequirement(\"87.248.113.14\", \"/cougarBaseline/v2/enumOperation\", \"BadRequest'}, {'tag': 'IP_ADDRESS', 'value': '87.248.113.14', 'start': 7065, 'end': 7078, 'context': 'eration\", \"BadRequest\"),new AccessLogRequirement(\"87.248.113.14\", \"/cougarBaseline/v2/enumOperation\", \"BadRequest'}, {'tag': 'IP_ADDRESS', 'value': '87.248.113.14', 'start': 6973, 'end': 6986}]\n",
      "\n",
      "new pii for 189: [{'tag': 'IP_ADDRESS', 'value': 'FF02::FB', 'start': 3326, 'end': 3334}, {'tag': 'IP_ADDRESS', 'value': '224.0.0.251', 'start': 3536, 'end': 3547}]\n",
      "\n",
      "new pii for 214: [{'tag': 'IP_ADDRESS', 'value': '127.0.0.1', 'start': 293, 'end': 302}]\n",
      "\n",
      "new pii for 254: [{'tag': 'USERNAME', 'value': 'denisbrodbeck', 'start': 45, 'end': 58, 'context': 'package logger\\n\\nimport (\\n\\t\"fmt\"\\n\\t\"github.com/denisbrodbeck/machineid\"\\n\\t\"io/ioutil\"\\n\\t\"net\"\\n\\t\"os\"\\n\\t\"path\"\\n\\t\"ru'}, {'tag': 'IP_ADDRESS', 'value': '192.168.0.9', 'start': 1514, 'end': 1525}]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "new_ds_2 = new_ds.map(update_pii_2, load_from_cache_file=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc2b4169e74549f3953b07884d2a3af5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3723704"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ds_2.to_json(\"data_final.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#more filtering\n",
    "res45 = \"\"\"'FP': [{'tag': 'IP_ADDRESS', 'value': '1.2.10.1', 'start': 31, 'end': 39}\"\"\"\n",
    "res161 = \"\"\"'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '47.04.66.11',\n",
    "      'start': 855,\n",
    "      'end': 866},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '42.06.16.1', 'start': 900, 'end': 910},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '73.27.21.6', 'start': 1136, 'end': 1146},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '98.53.38.14', 'start': 1150, 'end': 1161},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.27.36.39', 'start': 1168, 'end': 1178},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '13.1.22.22', 'start': 1208, 'end': 1218},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '7.93.31.27', 'start': 1413, 'end': 1423},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.15.66.46', 'start': 1430, 'end': 1440},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.58.25.53', 'start': 1450, 'end': 1460},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '2.39.19.66', 'start': 1745, 'end': 1755},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.6.34.41', 'start': 1766, 'end': 1775},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.23.89.48', 'start': 1782, 'end': 1792},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '1.59.28.59', 'start': 1804, 'end': 1814},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '58.06.81.17', 'start': 2235, 'end': 2246},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '11.44.29.6', 'start': 2250, 'end': 2260},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '99.08.41.13',\n",
    "      'start': 2279,\n",
    "      'end': 2290}]}\"\"\"\n",
    "\n",
    "res297 = \"\"\"'FP': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '20.0.0.2',\n",
    "      'start': 27297,\n",
    "      'end': 27305},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '60c0:a800::6',\n",
    "      'start': 27337,\n",
    "      'end': 27349},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '20.0.0.7', 'start': 27771, 'end': 27779},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '60c0:a800::7',\n",
    "      'start': 27811,\n",
    "      'end': 27823},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '20.0.0.3', 'start': 28238, 'end': 28246},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '60c0:a800::4',\n",
    "      'start': 28278,\n",
    "      'end': 28290},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '20.0.0.6', 'start': 29843, 'end': 29851},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '60c0:a800::3',\n",
    "      'start': 29883,\n",
    "      'end': 29895},\n",
    "     {'tag': 'IP_ADDRESS', 'value': '20.0.0.4', 'start': 30324, 'end': 30332},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '60c0:a800::8',\n",
    "      'start': 30364,\n",
    "      'end': 30376}]},\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['content', 'language', 'license', 'path', 'annotation_id', 'pii', 'pii_modified', 'id'],\n",
       "    num_rows: 400\n",
       "})"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ds_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False, ip_version)\n",
      "        counters_sanity_check.append(1)\n",
      "\n",
      "    def test_rules_priority_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we respect rule priorites in the forwarding case.\"\"\"\n",
      "        src_ip = \"20.0.0.7\" if ip_version == \"ipv4\" else \"60c0:a800::7\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(20)\n",
      "\n",
      "    def test_rules_priority_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we respect rule priorites in the drop case.\"\"\"\n",
      "        src_ip = \"20.0.0.3\" if ip_version == \"ipv4\" else \"60c0:a800::4\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(7)\n",
      "\n",
      "    def test_dest_ip_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward a packet on destination IP.\"\"\"\n",
      "        dst_ip = DOWNSTREAM_IP_TO_ALLOW[ip_version] if direction == \"uplink->downlink\" else UPSTREAM_IP_TO_ALLOW[ip_version]\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dst_ip=dst_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(2 if direction == \"uplink->downlink\" else 3)\n",
      "\n",
      "    def test_dest_ip_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop a packet on destination IP.\"\"\"\n",
      "        dst_ip = DOWNSTREAM_IP_TO_BLOCK[ip_version] if direction == \"uplink->downlink\" else UPSTREAM_IP_TO_BLOCK[ip_version]\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dst_ip=dst_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(15 if direction == \"uplink->downlink\" else 16)\n",
      "\n",
      "    def test_source_ip_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop a packet on source IP.\"\"\"\n",
      "        src_ip = \"20.0.0.6\" if ip_version == \"ipv4\" else \"60c0:a800::3\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(14)\n",
      "\n",
      "    def test_udp_source_ip_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward a UDP packet on source IP.\"\"\"\n",
      "        src_ip = \"20.0.0.4\" if ip_version == \"ipv4\" else \"60c0:a800::8\"\n",
      "        pkt = self.udp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(13)\n",
      "\n",
      "    def test_udp_source_ip_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop a UDP packet on source IP.\"\"\"\n",
      "        src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\n",
      "        pkt = self.udp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(26)\n",
      "\n",
      "    def test_icmp_source_ip_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop an ICMP packet on source IP.\"\"\"\n",
      "        src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\n",
      "        pkt = self.icmp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(25)\n",
      "\n",
      "    def test_icmp_source_ip_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward an ICMP packet on source IP.\"\"\"\n",
      "        src_ip = \"20.0.0.4\" if ip_version == \"ipv4\" else \"60c0:a800::8\"\n",
      "        pkt = self.icmp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(12)\n",
      "\n",
      "    def test_l4_dport_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on L4 destination port.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dport=0x1217)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(5)\n",
      "\n",
      "    def test_l4_sport_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on L4 source port.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, sport=0x120D)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(4)\n",
      "\n",
      "    def test_l4_dport_range_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on a range of L4 destination ports.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dport=0x123B)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(11)\n",
      "\n",
      "    def test_l4_sport_range_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on a range of L4 source ports.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, sport=0x123A)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(10)\n",
      "\n",
      "    def test_l4_dport_range_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on a range of L4 destination ports.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dport=0x127B)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(22)\n",
      "\n",
      "    def test_l4_sport_range_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on a range of L4 source ports.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, sport=0x1271)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(17)\n",
      "\n",
      "    def test_ip_proto_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on the IP protocol.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, proto=0x7E)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(5)\n",
      "\n",
      "    def test_tcp_flags_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and forward on the TCP flags.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, flags=0x1B)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(6)\n",
      "\n",
      "    def test_l4_dport_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on L4 destination port.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, dport=0x127B)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(22)\n",
      "\n",
      "    def test_l4_sport_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on L4 source port.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, sport=0x1271)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(10)\n",
      "\n",
      "    def test_ip_proto_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on the IP protocol.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, proto=0x7F)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(18)\n",
      "\n",
      "    def test_tcp_flags_match_dropped(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on the TCP flags.\"\"\"\n",
      "        pkt = self.tcp_packet(setup, direction, ptfadapter, ip_version, flags=0x24)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, True, ip_version)\n",
      "        counters_sanity_check.append(5)\n",
      "\n",
      "    def test_icmp_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check, ip_version):\n",
      "        \"\"\"Verify that we can match and drop on the TCP flags.\"\"\"\n",
      "        src_ip = \"20.0.0.10\" if ip_version == \"ipv4\" else \"60c0:a800::10\"\n",
      "        pkt = self.icmp_packet(setup, direction, ptfadapter, ip_version, src_ip=src_ip, icmp_type=3, icmp_code=1)\n",
      "\n",
      "        self._verify_acl_traffic(setup, direction, ptfadapter, pkt, False, ip_version)\n",
      "        counters_sanity_check.append(29)\n",
      "\n",
      "    def _verify_acl_traffic(self, setup, direction, ptfadapter, pkt, dropped, ip_version):\n",
      "        exp_pkt = self.expected_mask_routed_packet(pkt, ip_version)\n",
      "\n",
      "        if ip_version == \"ipv4\":\n",
      "            downstream_dst_port = DOWNSTREAM_IP_PORT_MAP.get(pkt[packet.IP].dst)\n",
      "        else:\n",
      "            downstream_dst_port = DOWNSTREAM_IP_PORT_MAP.get(pkt[packet.IPv6].dst)\n",
      "        ptfadapter.dataplane.flush()\n",
      "        testutils.send(ptfadapter, self.src_port, pkt)\n",
      "        if direction == \"uplink->downlink\" and downstream_dst_port:\n",
      "            if dropped:\n",
      "                testutils.verify_no_packet(ptfadapter, exp_pkt, downstream_dst_port)\n",
      "            else:\n",
      "                testutils.verify_packet(ptfadapter, exp_pkt, downstream_dst_port)\n",
      "        else:\n",
      "            if dropped:\n",
      "                testutils.verify_no_packet_any(ptfadapter, exp_pkt, ports=self.get_dst_ports(setup, direction))\n",
      "            else:\n",
      "                testutils.verify_packet_any_port(ptfadapter, exp_pkt, ports=self.get_dst_ports(setup, direction),\n",
      "                                                 timeout=20)\n",
      "\n",
      "\n",
      "class TestBasicAcl(BaseAclTest):\n",
      "    \"\"\"Test Basic functionality of ACL rules (i.e. setup with full update on a running device).\"\"\"\n",
      "\n",
      "    def setup_rules(self, dut, acl_table, ip_version):\n",
      "        \"\"\"Setup ACL rules for testing.\n",
      "\n",
      "        Args:\n",
      "            dut: The DUT having ACLs applied.\n",
      "            acl_table: Configuration info for the ACL table.\n",
      "\n",
      "        \"\"\"\n",
      "        table_name = acl_table[\"table_name\"]\n",
      "        dut.host.options[\"variable_manager\"].extra_vars.update({\"acl_table_name\": table_name})\n",
      "\n",
      "        logger.info(\"Generating basic ACL rules config for ACL table \\\"{}\\\" on {}\".format(table_name, dut))\n",
      "\n",
      "        dut_conf_file_path = os.path.join(DUT_TMP_DIR, \"acl_rules_{}.json\".format(table_name))\n",
      "        dut.template(src=os.path.join(TEMPLATE_DIR, ACL_RULES_FULL_TEMPLATE[ip_version]),\n",
      "                     dest=dut_conf_file_path)\n",
      "\n",
      "        logger.info(\"Applying ACL rules config \\\"{}\\\"\".format(dut_conf_file_path))\n",
      "        dut.command(\"config acl update full {}\".format(dut_conf_file_path))\n",
      "\n",
      "\n",
      "class TestIncrementalAcl(BaseAclTest):\n",
      "    \"\"\"Test ACL rule functionality with an incremental configuration.\n",
      "\n",
      "    Verify that everything still works as expected when an ACL configuration is applied in\n",
      "    multiple parts.\n",
      "    \"\"\"\n",
      "\n",
      "    def setup_rules(self, dut, acl_table, ip_version):\n",
      "        \"\"\"Setup ACL rules for testing.\n",
      "\n",
      "        Args:\n",
      "            dut: The DUT having ACLs applied.\n",
      "            acl_table: Configuration info for the ACL table.\n",
      "\n",
      "        \"\"\"\n",
      "        table_name = acl_table[\"table_name\"]\n",
      "        dut.host.options[\"variable_manager\"].extra_vars.update({\"acl_table_name\": table_name})\n",
      "\n",
      "        logger.info(\"Generating incremental ACL rules config for ACL table \\\"{}\\\"\"\n",
      "                    .format(table_name))\n",
      "\n",
      "        for part, config_file in enumerate(ACL_RULES_PART_TEMPLATES[ip_version]):\n",
      "            dut_conf_file_path = os.path.join(DUT_TMP_DIR, \"acl_rules_{}_part_{}.json\".format(table_name, part))\n",
      "            dut.template(src=os.path.join(TEMPLATE_DIR, config_file), dest=dut_conf_file_path)\n",
      "\n",
      "            logger.info(\"Applying ACL rules config \\\"{}\\\"\".format(dut_conf_file_path))\n",
      "            dut.command(\"config acl update incremental {}\".format(dut_conf_file_path))\n",
      "\n",
      "\n",
      "@pytest.mark.reboot\n",
      "class TestAclWithReboot(TestBasicAcl):\n",
      "    \"\"\"Test ACL rule functionality with a reboot.\n",
      "\n",
      "    Verify that configuration persists correctly after reboot and is applied properly\n",
      "    upon startup.\n",
      "    \"\"\"\n",
      "\n",
      "    def post_setup_hook(self, dut, localhost, populate_vlan_arp_entries, tbinfo):\n",
      "        \"\"\"Save configuration and reboot after rules are applied.\n",
      "\n",
      "        Args:\n",
      "            dut: The DUT having ACLs applied.\n",
      "            localhost: The host from which tests are run.\n",
      "            populate_vlan_arp_entries: A fixture to populate ARP/FDB tables for VLAN interfaces.\n",
      "\n",
      "        \"\"\"\n",
      "        dut.command(\"config save -y\")\n",
      "        reboot(dut, localhost, wait=240)\n",
      "        # We need some additional delay on e1031\n",
      "        if dut.facts[\"platform\"] == \"x86_64-cel_e1031-r0\":\n",
      "            time.sleep(240)\n",
      "        populate_vlan_arp_entries()\n",
      "\n",
      "\n",
      "@pytest.mark.port_toggle\n",
      "class TestAclWithPortToggle(TestBasicAcl):\n",
      "    \"\"\"Test ACL rule functionality after toggling ports.\n",
      "\n",
      "    Verify that ACLs still function as expected after links flap.\n",
      "    \"\"\"\n",
      "\n",
      "    def post_setup_hook(self, dut, localhost, populate_vlan_arp_entries, tbinfo):\n",
      "        \"\"\"Toggle ports after rules are applied.\n",
      "\n",
      "        Args:\n",
      "            dut: The DUT having ACLs applied.\n",
      "            localhost: The host from which tests are run.\n",
      "            populate_vlan_arp_entries: A fixture to populate ARP/FDB tables for VLAN interfaces.\n",
      "\n",
      "        \"\"\"\n",
      "        port_toggle(dut, tbinfo)\n",
      "        populate_vlan_arp_entries()\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(new_ds_2[\"content\"][297][27507:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "res52 = \"\"\"    'FN': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '0.0.0.0',\n",
    "      'start': 42191,\n",
    "      'end': 42198,\n",
    "      'context': 'ev_version.IsValid())\\n    prev_version = Version(\"0.0.0.0\");\\n  Version current_version(version_info.Version'}],\"\"\"\n",
    "\n",
    "res89 = \"\"\"    'FN': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '4.0.0.0',\n",
    "      'start': 942,\n",
    "      'end': 949,\n",
    "      'context': 'm.Resources.Tools.StronglyTypedResourceBuilder\", \"4.0.0.0\")]\\n    [global::System.Diagnostics.DebuggerNonUse'}]\"\"\"\n",
    "\n",
    "res129 = \"\"\"\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '1.1.1.1',\n",
    "      'start': 23855,\n",
    "      'end': 23862,\n",
    "      'context': 'ample\"\\n    primary     = true\\n    dns_servers = [\"1.1.1.1\", \"8.8.8.8\"]\\n\\n    ip_configuration {\\n      name  '},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '8.8.8.8',\n",
    "      'start': 34255,\n",
    "      'end': 34262,\n",
    "      'context': 'imary\"\\n    primary     = true\\n    dns_servers = [\"8.8.8.8\"]\\n\\n    ip_configuration {\\n      name      = \"inte'},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '8.8.8.8',\n",
    "      'start': 23866,\n",
    "      'end': 23873,\n",
    "      'context': 'primary     = true\\n    dns_servers = [\"1.1.1.1\", \"8.8.8.8\"]\\n\\n    ip_configuration {\\n      name      = \"inte'},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '8.8.8.8',\n",
    "      'start': 22803,\n",
    "      'end': 22810,\n",
    "      'context': 'ample\"\\n    primary     = true\\n    dns_servers = [\"8.8.8.8\"]\\n\\n    ip_configuration {\\n      name      = \"inte'}\"\"\"\n",
    "\n",
    "res150 = \"\"\"[{'tag': 'IP_ADDRESS',\n",
    "      'value': '4.0.0.0',\n",
    "      'start': 988,\n",
    "      'end': 995,\n",
    "      'context': 'm.Resources.Tools.StronglyTypedResourceBuilder\", \"4.0.0.0\")]\\n  [global::System.Diagnostics.DebuggerNonUserC'}]\"\"\"\n",
    "\n",
    "res327 = \"\"\"    'FN': [{'tag': 'IP_ADDRESS',\n",
    "      'value': '4.0.0.0',\n",
    "      'start': 2269,\n",
    "      'end': 2276,\n",
    "      'context': 'GeneratedCodeAttribute(\"PresentationBuildTasks\", \"4.0.0.0\")]\\n        public void InitializeComponent() {\\n  '},\n",
    "     {'tag': 'IP_ADDRESS',\n",
    "      'value': '4.0.0.0',\n",
    "      'start': 2920,\n",
    "      'end': 2927,\n",
    "      'context': 'GeneratedCodeAttribute(\"PresentationBuildTasks\", \"4.0.0.0\")]\\n        [System.ComponentModel.EditorBrowsable'}]\"\"\"\n",
    "\n",
    "res314 = \"\"\"    'FN': [{'tag': 'KEY',\n",
    "      'value': '12tvKAXCxZjSmdNbao16dKXC8tRWfcF5oc\\\\',\n",
    "      'start': 17731,\n",
    "      'end': 17766},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\',\n",
    "      'start': 9587,\n",
    "      'end': 9624},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1LtvqCaApEdUGFkpKMM4MstjcaL4dKg8SP\\\\\\\\\\\\',\n",
    "      'start': 9737,\n",
    "      'end': 9774},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\',\n",
    "      'start': 9544,\n",
    "      'end': 9581},\n",
    "     {'tag': 'KEY',\n",
    "      'value': '1PGFqEzfmQch1gKD3ra4k18PNj3tTUUSqg\\\\\\\\\\\\',\n",
    "      'start': 9694,\n",
    "      'end': 9731}]\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_pii_3(example):\n",
    "    from copy import deepcopy\n",
    "    new_example = deepcopy(example)\n",
    "    pii = json.loads(example[\"pii\"])\n",
    "    if example[\"id\"] == 297:\n",
    "        # add missing keys\n",
    "        key_1 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][27297:27305], \"start\": 27297, \"end\": 27305}\n",
    "        key_2 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][27337:27349], \"start\": 27337, \"end\": 27349}\n",
    "        key_3 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][27771:27779], \"start\": 27771, \"end\": 27779}\n",
    "        key_4 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][27811:27823], \"start\": 27811, \"end\": 27823}\n",
    "        key_5 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][28238:28246], \"start\": 28238, \"end\": 28246}\n",
    "        key_6 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][28278:28290], \"start\": 28278, \"end\": 28290}\n",
    "        key_7 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][29843:29851], \"start\": 29843, \"end\": 29851}\n",
    "        key_8 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][29883:29895], \"start\": 29883, \"end\": 29895}\n",
    "        key_9 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][30324:30332], \"start\": 30324, \"end\": 30332}\n",
    "        key_10 = {\"tag\": \"IP_ADDRESS\", \"value\": example[\"content\"][30364:30376], \"start\": 30364, \"end\": 30376}\n",
    "        pii.extend([key_1, key_2, key_3, key_4, key_5, key_6, key_7, key_8, key_9, key_10])\n",
    "        new_example[\"pii\"] = json.dumps(pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(pii)\n",
    "        print(f\"new pii for {example['id']}: {pii}\\n\")\n",
    "\n",
    "    elif example[\"id\"] == 52:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) != (42191, 42198)]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 89:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) != (942, 949)]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 129:\n",
    "        # remove all keys in there\n",
    "        indexes = [(23855, 23862), (34255, 34262), (23866, 23873), (22803, 22810)]\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) not in indexes]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 150:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) != (988, 995)]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 327:\n",
    "        # remove key\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) not in [(2269, 2276), (2920, 2927)]]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    \n",
    "    elif example[\"id\"] == 314:\n",
    "        # remove key\n",
    "        indexes = [(17731, 17766), (9587, 9624), (9737, 9774), (9544, 9581), (9694, 9731)]\n",
    "        new_pii = [key for key in pii if (key[\"start\"], key[\"end\"]) not in indexes]\n",
    "        new_example[\"pii\"] = json.dumps(new_pii)\n",
    "        new_example[\"pii_modified\"] = json.dumps(new_pii)\n",
    "        print(f\"new pii for {example['id']}: {new_pii}\\n\")\n",
    "    return new_example        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7a4b48a2ebd4b3c8574b3e0f4487a0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/400 [00:00<?, ?ex/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "new pii for 52: []\n",
      "\n",
      "new pii for 89: []\n",
      "\n",
      "new pii for 129: [{'tag': 'IP_ADDRESS', 'value': '10.0.0.0/16', 'start': 14710, 'end': 14721, 'context': 'ource_group.test.name}\"\\n  address_space       = [\"10.0.0.0/16\"]\\n  location            = \"${azurerm_resource_gro'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 27671, 'end': 27680, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 35135, 'end': 35148, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 41753, 'end': 41762, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'IP_ADDRESS', 'value': '10.0.0.0/24', 'start': 15018, 'end': 15029, 'context': 'ual_network.test.name}\"\\n  address_prefix       = \"10.0.0.0/24\"\\n}\\n\\nresource \"azurerm_public_ip\" \"test\" {\\n  name '}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 17980, 'end': 17989, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 30151, 'end': 30164, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 29021, 'end': 29034, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 36869, 'end': 36878, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 40659, 'end': 40672, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 43165, 'end': 43174, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 31224, 'end': 31233, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 23425, 'end': 23434, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 30115, 'end': 30124, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 32466, 'end': 32479, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 33825, 'end': 33834, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 31260, 'end': 31273, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 24519, 'end': 24532, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 13649, 'end': 13662, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 35099, 'end': 35108, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 39618, 'end': 39627, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 44363, 'end': 44376, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 24483, 'end': 24492, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 18016, 'end': 18029, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 41789, 'end': 41802, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 32430, 'end': 32439, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 25542, 'end': 25551, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 25578, 'end': 25591, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 43201, 'end': 43214, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 28985, 'end': 28994, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 38466, 'end': 38475, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 38502, 'end': 38515, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 44327, 'end': 44336, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 23461, 'end': 23474, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 27707, 'end': 27720, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 19543, 'end': 19552, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'IP_ADDRESS', 'value': '10.0.1.0/24', 'start': 17656, 'end': 17667, 'context': 'ual_network.test.name}\"\\n  address_prefix       = \"10.0.1.0/24\"\\n}\\n\\nresource \"azurerm_linux_virtual_machine_scale'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 19579, 'end': 19592, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 40623, 'end': 40632, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 13613, 'end': 13622, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'USERNAME', 'value': 'adminuser', 'start': 22373, 'end': 22382, 'context': ' instances           = 1\\n  admin_username      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disab'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 22409, 'end': 22422, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 33861, 'end': 33874, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 39654, 'end': 39667, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}, {'tag': 'PASSWORD', 'value': 'P@ssword1234!', 'start': 36905, 'end': 36918, 'context': 'rname      = \"adminuser\"\\n  admin_password      = \"P@ssword1234!\"\\n\\n  disable_password_authentication = false\\n\\n  so'}]\n",
      "\n",
      "new pii for 150: []\n",
      "\n",
      "new pii for 297: [{'tag': 'IP_ADDRESS', 'value': '192.168.136.1', 'start': 2476, 'end': 2489, 'context': ':a800::2\"\\n}\\nUPSTREAM_IP_TO_ALLOW = {\\n    \"ipv4\": \"192.168.136.1\",\\n    \"ipv6\": \"40c0:a800::4\"\\n}\\nUPSTREAM_IP_TO_BLO'}, {'tag': 'IP_ADDRESS', 'value': '20c0:a800::4', 'start': 2229, 'end': 2241, 'context': 'LOW = {\\n    \"ipv4\": \"192.168.0.252\",\\n    \"ipv6\": \"20c0:a800::4\"\\n}\\nDOWNSTREAM_IP_TO_BLOCK = {\\n    \"ipv4\": \"192.16'}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.8', 'start': 30801, 'end': 30809, 'context': 'p a UDP packet on source IP.\"\"\"\\n        src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\\n    '}, {'tag': 'IP_ADDRESS', 'value': '40c0:a800::2', 'start': 2422, 'end': 2434, 'context': '_IP = {\\n    \"ipv4\": \"192.168.128.1\",\\n    \"ipv6\": \"40c0:a800::2\"\\n}\\nUPSTREAM_IP_TO_ALLOW = {\\n    \"ipv4\": \"192.168.'}, {'tag': 'IP_ADDRESS', 'value': '192.168.128.1', 'start': 2393, 'end': 2406, 'context': 'P_PORT_MAP = {}\\n\\nUPSTREAM_DST_IP = {\\n    \"ipv4\": \"192.168.128.1\",\\n    \"ipv6\": \"40c0:a800::2\"\\n}\\nUPSTREAM_IP_TO_ALL'}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.8', 'start': 31280, 'end': 31288, 'context': 'an ICMP packet on source IP.\"\"\"\\n        src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\\n    '}, {'tag': 'IP_ADDRESS', 'value': '192.168.0.252', 'start': 2200, 'end': 2213, 'context': '800::2\"\\n}\\nDOWNSTREAM_IP_TO_ALLOW = {\\n    \"ipv4\": \"192.168.0.252\",\\n    \"ipv6\": \"20c0:a800::4\"\\n}\\nDOWNSTREAM_IP_TO_B'}, {'tag': 'IP_ADDRESS', 'value': '20c0:a800::8', 'start': 2314, 'end': 2326, 'context': 'OCK = {\\n    \"ipv4\": \"192.168.0.251\",\\n    \"ipv6\": \"20c0:a800::8\"\\n}\\n\\nDOWNSTREAM_IP_PORT_MAP = {}\\n\\nUPSTREAM_DST_IP '}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::10', 'start': 37065, 'end': 37078, 'context': 'rc_ip = \"20.0.0.10\" if ip_version == \"ipv4\" else \"60c0:a800::10\"\\n        pkt = self.icmp_packet(setup, direction,'}, {'tag': 'IP_ADDRESS', 'value': '40c0:a800::4', 'start': 2505, 'end': 2517, 'context': 'LOW = {\\n    \"ipv4\": \"192.168.136.1\",\\n    \"ipv6\": \"40c0:a800::4\"\\n}\\nUPSTREAM_IP_TO_BLOCK = {\\n    \"ipv4\": \"192.168.'}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::2', 'start': 30841, 'end': 30853, 'context': 'src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\\n        pkt = self.udp_packet(setup, direction, '}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::2', 'start': 31320, 'end': 31332, 'context': 'src_ip = \"20.0.0.8\" if ip_version == \"ipv4\" else \"60c0:a800::2\"\\n        pkt = self.icmp_packet(setup, direction,'}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.4', 'start': 31765, 'end': 31773, 'context': 'an ICMP packet on source IP.\"\"\"\\n        src_ip = \"20.0.0.4\" if ip_version == \"ipv4\" else \"60c0:a800::8\"\\n    '}, {'tag': 'IP_ADDRESS', 'value': '192.168.144.1', 'start': 2559, 'end': 2572, 'context': ':a800::4\"\\n}\\nUPSTREAM_IP_TO_BLOCK = {\\n    \"ipv4\": \"192.168.144.1\",\\n    \"ipv6\": \"40c0:a800::8\"\\n}\\n\\nVLAN_BASE_MAC_PAT'}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::5', 'start': 1918, 'end': 1930, 'context': 'T_SRC_IP = {\\n    \"ipv4\": \"20.0.0.1\",\\n    \"ipv6\": \"60c0:a800::5\"\\n}\\n\\n\\n# TODO: These routes don\\'t match the VLAN in'}, {'tag': 'IP_ADDRESS', 'value': '192.168.0.253', 'start': 2115, 'end': 2128, 'context': '6 tests for T0\\nDOWNSTREAM_DST_IP = {\\n    \"ipv4\": \"192.168.0.253\",\\n    \"ipv6\": \"20c0:a800::2\"\\n}\\nDOWNSTREAM_IP_TO_A'}, {'tag': 'IP_ADDRESS', 'value': '192.168.0.251', 'start': 2285, 'end': 2298, 'context': '800::4\"\\n}\\nDOWNSTREAM_IP_TO_BLOCK = {\\n    \"ipv4\": \"192.168.0.251\",\\n    \"ipv6\": \"20c0:a800::8\"\\n}\\n\\nDOWNSTREAM_IP_POR'}, {'tag': 'IP_ADDRESS', 'value': '40c0:a800::8', 'start': 2588, 'end': 2600, 'context': 'OCK = {\\n    \"ipv4\": \"192.168.144.1\",\\n    \"ipv6\": \"40c0:a800::8\"\\n}\\n\\nVLAN_BASE_MAC_PATTERN = \"72060001{:04}\"\\n\\nLOG_'}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.1', 'start': 1894, 'end': 1902, 'context': ' xrange(1, 3))\\n}\\n\\nDEFAULT_SRC_IP = {\\n    \"ipv4\": \"20.0.0.1\",\\n    \"ipv6\": \"60c0:a800::5\"\\n}\\n\\n\\n# TODO: These ro'}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::8', 'start': 31805, 'end': 31817, 'context': 'src_ip = \"20.0.0.4\" if ip_version == \"ipv4\" else \"60c0:a800::8\"\\n        pkt = self.icmp_packet(setup, direction,'}, {'tag': 'IP_ADDRESS', 'value': '20c0:a800::2', 'start': 2144, 'end': 2156, 'context': '_IP = {\\n    \"ipv4\": \"192.168.0.253\",\\n    \"ipv6\": \"20c0:a800::2\"\\n}\\nDOWNSTREAM_IP_TO_ALLOW = {\\n    \"ipv4\": \"192.16'}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.10', 'start': 37024, 'end': 37033, 'context': 'h and drop on the TCP flags.\"\"\"\\n        src_ip = \"20.0.0.10\" if ip_version == \"ipv4\" else \"60c0:a800::10\"\\n   '}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.2', 'start': 27297, 'end': 27305}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::6', 'start': 27337, 'end': 27349}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.7', 'start': 27771, 'end': 27779}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::7', 'start': 27811, 'end': 27823}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.3', 'start': 28238, 'end': 28246}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::4', 'start': 28278, 'end': 28290}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.6', 'start': 29843, 'end': 29851}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::3', 'start': 29883, 'end': 29895}, {'tag': 'IP_ADDRESS', 'value': '20.0.0.4', 'start': 30324, 'end': 30332}, {'tag': 'IP_ADDRESS', 'value': '60c0:a800::8', 'start': 30364, 'end': 30376}]\n",
      "\n",
      "new pii for 314: [{'tag': 'NAME', 'value': 'Satoshi Nakamoto', 'start': 22, 'end': 38, 'context': '// Copyright (c) 2010 Satoshi Nakamoto\\n// Original Code: Copyright (c) 2009-2014 The Bit'}]\n",
      "\n",
      "new pii for 327: []\n",
      "\n"
     ]
    }
   ],
   "source": [
    "new_ds_3 = new_ds_2.map(update_pii_3, load_from_cache_file=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3dcc4db6f2a646169ffaedab4d88e86b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "3720260"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_ds_3.to_json(\"new_ds.json\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 ('venv')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "fd8fde6f83dada9276d12fdb71d773558994168ed1b3bea457b8db38c02aa2e1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
